Oct 30 18:38:08.969477 ip-10-0-143-97 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:38:09.379911 ip-10-0-143-97 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:38:09.379911 ip-10-0-143-97 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:38:09.379911 ip-10-0-143-97 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:38:09.379911 ip-10-0-143-97 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:38:09.384832 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384715    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:09.384832 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384832    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:09.384832 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384836    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384840    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384843    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384846    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384849    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384852    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384855    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384857    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384860    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384862    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384865    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384868    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384871    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384875    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384879    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384882    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384885    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384888    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384891    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:09.384927 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384894    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384897    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384900    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384903    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384905    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384908    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384910    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384913    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384916    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384918    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384921    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384924    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384926    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384929    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384931    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384935    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384938    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384941    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384943    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384946    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:09.385385 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384948    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384951    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384953    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384956    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384959    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384961    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384964    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384966    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384968    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384971    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384973    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384976    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384978    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384980    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384985    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384987    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384990    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384992    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384995    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:09.385952 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.384997    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385000    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385002    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385004    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385007    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385009    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385013    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385016    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385019    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385022    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385025    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385027    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385030    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385033    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385035    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385038    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385040    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385043    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385045    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:09.386416 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385048    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385050    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385052    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385055    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385058    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385060    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385063    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385065    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385068    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385072    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385074    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385076    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385079    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385083    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385085    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385088    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385090    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385093    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385095    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385098    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:09.386885 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385100    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385103    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385105    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385108    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385110    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385113    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385115    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385955    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385964    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385968    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385971    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385973    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385976    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385979    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385981    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385984    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385987    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385990    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385992    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:09.387369 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385995    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.385998    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386001    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386004    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386007    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386010    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386012    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386015    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386032    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386036    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386040    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386043    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386046    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386049    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386051    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386054    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386057    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386059    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386062    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:09.387842 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386064    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386067    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386069    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386072    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386074    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386077    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386079    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386082    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386084    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386088    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386090    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386093    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386095    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386098    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386101    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386104    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386107    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386110    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386112    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386115    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:09.388319 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386118    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386120    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386123    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386125    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386128    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386130    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386133    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386147    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386150    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386153    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386155    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386158    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386161    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386163    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386166    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386168    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386171    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386173    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386176    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386178    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:09.388791 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386182    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386184    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386187    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386189    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386192    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386195    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386198    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386201    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386204    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386206    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386210    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386212    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386214    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386217    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386219    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386222    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386224    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386226    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386229    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386231    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:09.389284 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386234    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386236    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386239    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386242    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386244    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386247    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386249    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386252    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386254    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386256    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386259    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386261    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386264    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386266    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.386269    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387608    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387625    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387629    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387633    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387636    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:09.389797 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387639    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387642    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387645    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387648    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387651    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387653    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387657    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387660    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387663    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387666    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387668    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387670    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387673    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387676    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387679    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387681    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387683    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387686    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387689    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387691    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:09.390652 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387694    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387697    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387699    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387702    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387704    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387707    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387709    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387712    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387714    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387716    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387719    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387722    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387724    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387727    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387729    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387732    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387735    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387737    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387739    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:09.391394 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387742    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387744    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387747    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387749    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387752    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387754    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387757    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387759    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387762    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387764    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387766    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387769    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387771    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387774    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387776    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387779    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387781    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387783    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387786    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:09.391854 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387788    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387790    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387793    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387795    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387800    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387803    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387805    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387808    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387812    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387815    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387818    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387821    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387823    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387825    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387828    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387831    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387833    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387836    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387838    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387841    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:09.392339 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387844    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387847    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387849    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387852    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387854    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387857    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387860    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387862    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387865    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387867    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387870    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387872    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387875    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387877    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387880    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387882    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387886    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387888    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387891    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:09.392815 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387893    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387896    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387899    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.387902    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389022    2578 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389037    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389044    2578 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389049    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389053    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389057    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389062    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389066    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389070    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389072    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389076    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389079    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389082    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389085    2578 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389088    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389091    2578 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389094    2578 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389097    2578 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389101    2578 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389104    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:38:09.393329 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389107    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389110    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389113    2578 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389117    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389120    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389123    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389127    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389130    2578 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389133    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389148    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389152    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389155    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389159    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389162    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389165    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389169    2578 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389172    2578 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389174    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389179    2578 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389182    2578 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389185    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389188    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389190    2578 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389195    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:38:09.393896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389198    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389201    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389204    2578 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389207    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389210    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389213    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389216    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389219    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389222    2578 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389224    2578 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389229    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389232    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389235    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389238    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389241    2578 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389244    2578 flags.go:64] FLAG: --help="false"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389247    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389250    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389253    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389256    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389259    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389263    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389267    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:38:09.394533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389271    2578 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389274    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389277    2578 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389280    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389283    2578 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389286    2578 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389289    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389292    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389295    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389298    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389301    2578 flags.go:64] FLAG: --lock-file=""
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389304    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389306    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389309    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389315    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389318    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389322    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389325    2578 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389327    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389331    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389333    2578 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389337    2578 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389342    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389345    2578 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389348    2578 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:38:09.395099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389351    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389354    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389358    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389360    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389364    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389366    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389369    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389378    2578 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389381    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389384    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389387    2578 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389390    2578 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389393    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389396    2578 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389398    2578 flags.go:64] FLAG: --port="10250"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389401    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389404    2578 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-093c7716fc2eeca41"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389407    2578 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389410    2578 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389413    2578 flags.go:64] FLAG: --register-node="true"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389415    2578 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389419    2578 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389422    2578 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389424    2578 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389427    2578 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389431    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:38:09.395712 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389434    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389437    2578 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389439    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389442    2578 flags.go:64] FLAG: --runonce="false"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389445    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389448    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389451    2578 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389454    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389456    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389460    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389463    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389468    2578 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389471    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389474    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389477    2578 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389480    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389483    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389486    2578 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389489    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389495    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389498    2578 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389501    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389505    2578 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389508    2578 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389511    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:38:09.396356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389514    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389517    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389520    2578 flags.go:64] FLAG: --v="2"
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389524    2578 flags.go:64] FLAG: --version="false"
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389528    2578 flags.go:64] FLAG: --vmodule=""
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389532    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.389535    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.389983    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.389997    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390003    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390011    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390016    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390028    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390032    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390036    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390040    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390045    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390049    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390055    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390059    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390063    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:09.396995 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390067    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390073    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390077    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390086    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390091    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390095    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390099    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390104    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390108    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390112    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390115    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390119    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390123    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390128    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390132    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390154    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390165    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390170    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390174    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:09.397543 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390179    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390183    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390190    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390195    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390202    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390208    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390213    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390217    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390221    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390232    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390238    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390242    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390246    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390250    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390255    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390260    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390264    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390268    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390272    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:09.398002 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390277    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390281    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390285    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390295    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390299    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390304    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390308    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390312    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390316    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390321    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390327    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390333    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390337    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390341    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390346    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390358    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390362    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390367    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390371    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:09.398471 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390375    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390380    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390384    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390388    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390393    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390397    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390401    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390406    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390411    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390420    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390425    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390430    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390434    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390438    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390442    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390445    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390450    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390454    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390458    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390462    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:09.398918 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390467    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390471    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390480    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390484    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390488    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390492    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390496    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390500    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390506    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390510    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390514    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390518    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390522    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390527    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390531    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.390597    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390719    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:09.399411 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390728    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390733    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390737    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390742    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390746    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390750    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390755    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390766    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390770    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390774    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390778    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390782    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390786    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390791    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390795    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390799    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390803    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390807    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390812    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:09.399807 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390820    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390825    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390828    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390833    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390838    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390842    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390846    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390850    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390854    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390858    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390862    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390866    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390870    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390880    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390887    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390899    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390904    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390906    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390909    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:09.400278 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390912    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390916    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390919    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390923    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390925    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.390935    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391072    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391075    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391078    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391081    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391086    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391088    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391091    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391094    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391097    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391099    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391102    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391125    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391129    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:09.400730 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391133    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391152    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391156    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391159    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391162    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391165    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391169    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391173    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391176    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391179    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391181    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391184    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391186    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391189    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391191    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391194    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391196    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391199    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391201    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:09.401206 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391204    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391206    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391209    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391211    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391214    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391216    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391219    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391222    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391225    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391227    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391232    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391238    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391240    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391243    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391246    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391249    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391252    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391254    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391257    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391259    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:09.401657 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391264    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391267    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391270    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391272    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391275    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391277    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391280    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391282    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.391285    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.397960    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.397978    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.398787    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.398793    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398854    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398859    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398862    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398865    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398868    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398871    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398873    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:09.402275 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398876    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398879    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398881    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398884    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398887    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398890    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398892    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398896    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398899    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398902    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398904    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398907    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398910    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398912    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398915    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398918    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398921    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398924    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398927    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:09.402738 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398929    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398932    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398935    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398939    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398943    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398947    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398949    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398952    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398955    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398958    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398960    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398962    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398965    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398968    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398970    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398973    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398976    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398978    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398981    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:09.403198 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398983    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398986    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398989    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398992    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398994    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398997    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.398999    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399003    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399006    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399010    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399013    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399016    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399018    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399021    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399024    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399027    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399030    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399032    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399035    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:09.403678 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399037    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399040    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399043    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399045    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399048    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399050    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399053    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399056    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399058    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399061    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399063    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399066    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399068    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399071    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399074    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399076    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399079    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399082    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399085    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399087    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:09.404128 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399090    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399093    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399095    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399098    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399101    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399103    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399106    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399108    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399111    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399114    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399116    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399119    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399121    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399124    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399126    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399129    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399131    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399134    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399151    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:09.404615 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399155    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399158    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399160    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.399189    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399335    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399339    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399343    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399346    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399348    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399351    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399354    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399358    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399362    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399366    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399369    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:09.405061 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399372    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399375    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399377    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399380    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399383    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399385    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399388    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399391    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399393    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399395    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399398    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399400    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399403    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399405    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399408    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399411    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399414    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399416    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399419    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:09.405499 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399422    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399426    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399430    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399434    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399437    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399439    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399442    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399444    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399447    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399449    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399452    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399454    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399457    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399459    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399462    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399465    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399467    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399470    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399472    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:09.405946 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399475    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399477    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399479    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399482    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399485    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399487    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399489    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399492    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399494    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399517    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399521    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399524    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399527    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399530    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399533    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399535    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399538    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399541    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399543    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399546    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:09.406515 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399548    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399551    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399553    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399557    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399559    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399562    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399565    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399567    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399570    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399572    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399575    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399577    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399580    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399582    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399585    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399587    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399590    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399592    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399595    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:09.406983 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399597    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399600    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399602    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399604    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399607    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399609    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399612    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399614    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399617    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399620    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399622    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399625    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399627    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399630    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399632    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399635    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399637    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:09.407439 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399640    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.399663    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399706    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399709    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399712    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399715    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399719    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399721    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399726    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399729    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399732    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399734    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399737    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399739    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399741    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399744    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:09.407836 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399747    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399749    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399752    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399754    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399757    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399759    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399763    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399767    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399770    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399773    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399776    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399778    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399781    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399783    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399786    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399788    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399791    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399794    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399796    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:09.408233 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399799    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399801    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399804    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399806    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399809    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399813    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399815    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399818    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399821    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399824    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399827    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399829    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399831    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399834    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399836    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399839    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399841    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399843    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399846    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399848    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:09.408704 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399851    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399854    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399856    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399859    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399861    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399864    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399867    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399869    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399872    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399874    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399877    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399880    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399883    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399885    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399888    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399890    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399892    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399896    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399899    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399901    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:09.409196 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399903    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399906    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399908    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399911    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399913    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399916    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399918    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399920    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399923    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399925    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399928    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399930    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399932    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399935    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399938    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399940    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399942    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399945    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399947    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:09.409671 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399949    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399952    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399954    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399957    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399959    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399962    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399965    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399967    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399970    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399972    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399977    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399982    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399984    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:09.399987    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.400047    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.402847    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.403851    2578 server.go:1014] "Starting client certificate rotation"
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.403944    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:38:09.410129 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.403994    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:38:09.410634 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.405913    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:38:09.410661 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.410631    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:38:09.419322 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.419303    2578 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:38:09.422250 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.422236    2578 log.go:25] "Validated CRI v1 image API"
Oct 30 18:38:09.423049 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.423019    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:38:09.426949 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.426924    2578 fs.go:135] Filesystem UUIDs: map[130e84bc-1515-41f0-bf18-284563424445:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a3db4ba5-e65d-4323-a9d1-a2d700002524:/dev/nvme0n1p3]
Oct 30 18:38:09.427023 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.426948    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 18:38:09.432884 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.432763    2578 manager.go:217] Machine: {Timestamp:2026-10-30 18:38:09.431537916 +0000 UTC m=+0.342813372 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098942 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b2460964a3bc81d80f8d4cc5ef51d SystemUUID:ec2b2460-964a-3bc8-1d80-f8d4cc5ef51d BootID:ac3f7d52-8e75-4e4f-b991-8398ebb2412b Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:66:a3:39:f8:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:66:a3:39:f8:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:5a:ad:d3:30:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:38:09.433623 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.433611    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:38:09.433716 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.433704    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:38:09.436577 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.436544    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:38:09.436713 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.436579    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-97.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:38:09.436755 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.436716    2578 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:38:09.436755 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.436724    2578 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:38:09.436755 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.436753    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:38:09.436832 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.436772    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:38:09.437456 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.437446    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:38:09.439848 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.439836    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:38:09.439967 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.439958    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:38:09.440545 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.440535    2578 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:38:09.440575 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.440551    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:38:09.440575 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.440562    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:38:09.440575 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.440570    2578 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:38:09.440659 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.440578    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:38:09.440925 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.440913    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:38:09.443127 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.443110    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:38:09.447278 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.447266    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:38:09.448523 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.448512    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:38:09.448578 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.448542    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:38:09.448689 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.448679    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:38:09.448722 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.448695    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:38:09.448722 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.448702    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:38:09.448722 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.448707    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:38:09.448722 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.448713    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:38:09.448722 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.448719    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:38:09.449351 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.449342    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:38:09.449385 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.449353    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:38:09.449385 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.449359    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:38:09.449385 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.449370    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:38:09.449385 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.449383    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:38:09.449509 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.449392    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:38:09.450305 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.450296    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:38:09.450339 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.450306    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:38:09.454068 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.454056    2578 server.go:1263] "Started kubelet"
Oct 30 18:38:09.454219 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.454192    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:38:09.454284 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.454191    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:38:09.454284 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.454271    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:38:09.454792 ip-10-0-143-97 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:38:09.456433 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.456411    2578 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:38:09.457817 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.457797    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:38:09.460710 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:09.460693    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 18:38:09.461556 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.461539    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:38:09.462032 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.462015    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:38:09.462407 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:09.462383    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-97.us-west-1.compute.internal\" not found"
Oct 30 18:38:09.462627 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.462594    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:38:09.462627 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.462613    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:38:09.462763 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.462622    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:38:09.462763 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.462716    2578 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:38:09.462763 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.462726    2578 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:38:09.463082 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.463068    2578 factory.go:55] Registering systemd factory
Oct 30 18:38:09.463082 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.463083    2578 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:38:09.463315 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.463305    2578 factory.go:153] Registering CRI-O factory
Oct 30 18:38:09.463315 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.463316    2578 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:38:09.463384 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.463358    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:38:09.463384 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.463371    2578 factory.go:103] Registering Raw factory
Oct 30 18:38:09.463384 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.463382    2578 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:38:09.464240 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.464226    2578 manager.go:319] Starting recovery of all containers
Oct 30 18:38:09.475620 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.475603    2578 manager.go:324] Recovery completed
Oct 30 18:38:09.476996 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:09.476965    2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 18:38:09.480016 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.480003    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:09.481898 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.481878    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:09.481994 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.481916    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:09.481994 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.481927    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:09.482528 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.482510    2578 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:38:09.482528 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.482524    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:38:09.482644 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.482543    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:09.482644 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.482542    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:38:09.483277 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.483260    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:09.483347 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.483289    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:09.483347 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.483303    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:09.483806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.483795    2578 policy_none.go:50] "Start"
Oct 30 18:38:09.483847 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.483811    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:38:09.483847 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.483824    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:38:09.484664 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.484653    2578 policy_none.go:44] "Start"
Oct 30 18:38:09.535895 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.535715    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:38:09.551839 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.551820    2578 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:38:09.551959 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:09.551851    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:38:09.551959 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.551862    2578 server.go:87] "Starting device plugin registration server"
Oct 30 18:38:09.552107 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.552095    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:38:09.552184 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.552107    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:38:09.552250 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.552212    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:38:09.552298 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.552287    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:38:09.552298 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.552295    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:38:09.552874 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:09.552848    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:38:09.552985 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:09.552903    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-97.us-west-1.compute.internal\" not found"
Oct 30 18:38:09.598564 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.598497    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-97.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:38:09.609398 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.609366    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:38:09.610765 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.610748    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:38:09.610848 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.610771    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:38:09.611267 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.611251    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:38:09.611371 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:09.611298    2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 18:38:09.646950 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:09.646918    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-97.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 18:38:09.648010 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:09.646946    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-97.us-west-1.compute.internal.18e36299d43f586e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-97.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-97.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 18:38:09.45402891 +0000 UTC m=+0.365304366,LastTimestamp:2026-10-30 18:38:09.45402891 +0000 UTC m=+0.365304366,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-97.us-west-1.compute.internal,}"
Oct 30 18:38:09.652403 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.652386    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:09.653234 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.653209    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:09.653323 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.653245    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:09.653323 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.653260    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:09.653323 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.653294    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.676822 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.676796    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:38:09.676956 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.676840    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lzrhn"
Oct 30 18:38:09.712223 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.712183    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"]
Oct 30 18:38:09.712330 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.712268    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:09.713186 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.713166    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:09.713186 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.713201    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:09.713339 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.713217    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:09.714533 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.714520    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:09.714696 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.714681    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.714737 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.714713    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:09.715343 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.715327    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:09.715343 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.715338    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:09.715464 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.715356    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:09.715464 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.715367    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:09.715464 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.715357    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:09.715464 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.715433    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:09.716594 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.716579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.716673 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.716604    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:09.717393 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.717375    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:09.717475 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.717406    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:09.717475 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.717416    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:09.743549 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:09.743523    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-97.us-west-1.compute.internal\" not found" node="ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.747041 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:09.747022    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-97.us-west-1.compute.internal\" not found" node="ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.755320 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.755293    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:09.756795 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.756777    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:09.764003 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.763979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a238862e2cf33c9dc387d53f60c786c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"5a238862e2cf33c9dc387d53f60c786c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.764093 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.764008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/349d28f3fc367c9ffa500d3d4a12aae5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"349d28f3fc367c9ffa500d3d4a12aae5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.764093 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.764031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/349d28f3fc367c9ffa500d3d4a12aae5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"349d28f3fc367c9ffa500d3d4a12aae5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.852966 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.852885    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:09.853857 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.853837    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:09.854168 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.854151    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.854237 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:09.854184    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-97.us-west-1.compute.internal\": node \"ip-10-0-143-97.us-west-1.compute.internal\" not found"
Oct 30 18:38:09.862357 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.862336    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.864168 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.864133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a238862e2cf33c9dc387d53f60c786c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"5a238862e2cf33c9dc387d53f60c786c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.864230 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.864178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/349d28f3fc367c9ffa500d3d4a12aae5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"349d28f3fc367c9ffa500d3d4a12aae5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.864230 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.864198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/349d28f3fc367c9ffa500d3d4a12aae5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"349d28f3fc367c9ffa500d3d4a12aae5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.864324 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.864252    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a238862e2cf33c9dc387d53f60c786c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"5a238862e2cf33c9dc387d53f60c786c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.864324 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.864256    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/349d28f3fc367c9ffa500d3d4a12aae5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"349d28f3fc367c9ffa500d3d4a12aae5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.864324 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.864256    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/349d28f3fc367c9ffa500d3d4a12aae5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"349d28f3fc367c9ffa500d3d4a12aae5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:09.974650 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.974621    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:09.974807 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:09.974726    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:10.031649 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.031623    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:10.046829 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.046801    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:10.050453 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.050286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:38:10.404016 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.403979    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 18:38:10.404658 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.404202    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:38:10.404658 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.404228    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:38:10.404658 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.404225    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:38:10.404658 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:10.404222    2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aec3e5e704cd849e78ca3402ce7e1bb4-9d227f372467f59d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-97.us-west-1.compute.internal.18e36299d5e917d2\": read tcp 10.0.143.97:58544->3.128.161.143:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-97.us-west-1.compute.internal.18e36299d5e917d2  default   5355 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-97.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-97.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-97.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 18:38:09 +0000 UTC,LastTimestamp:2026-10-30 18:38:09.653266222 +0000 UTC m=+0.564541682,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-97.us-west-1.compute.internal,}"
Oct 30 18:38:10.441233 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.441198    2578 apiserver.go:52] "Watching apiserver"
Oct 30 18:38:10.462045 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.462020    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 18:38:10.680462 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.680373    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:33:09 +0000 UTC" deadline="2028-08-11 19:36:26.166945664 +0000 UTC"
Oct 30 18:38:10.680462 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.680404    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15624h58m15.486544641s"
Oct 30 18:38:10.710736 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.710719    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:38:10.711012 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.710993    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-wwvr8","kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal","openshift-image-registry/node-ca-8pwf7","openshift-multus/network-metrics-daemon-676kr","openshift-cluster-node-tuning-operator/tuned-plh42","openshift-dns/node-resolver-7mfxz","openshift-multus/multus-additional-cni-plugins-6bjz2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-27m75","openshift-network-operator/iptables-alerter-ctrbm","kube-system/global-pull-secret-syncer-7h2cm","openshift-multus/multus-pkq65","openshift-ovn-kubernetes/ovnkube-node-54qsj"]
Oct 30 18:38:10.712372 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.712358    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.713341 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.713323    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.714301 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.714286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:38:10.715212 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.715199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:10.715280 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:10.715257    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:10.716106 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.716094    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.718557 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.718316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.718818 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.718803    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:38:10.719715 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.719700    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:10.719772 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:10.719752    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:10.720666 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.720651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:38:10.721616 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.721603    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:10.721676 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:10.721656    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:10.722646 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.722633    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:38:10.723808 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.723791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.764198 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.764170    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:38:10.764854 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.764835    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:38:10.764961 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.764874    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:38:10.764961 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.764904    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:38:10.764961 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.764956    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:38:10.765132 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.765010    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gk56f\""
Oct 30 18:38:10.765132 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.765019    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:38:10.765132 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.765027    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:38:10.765354 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.765340    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nsz9w\""
Oct 30 18:38:10.765418 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.765339    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:38:10.765470 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.765440    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:38:10.765470 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.765462    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:38:10.765585 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.765535    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kbv4s\""
Oct 30 18:38:10.767250 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.767234    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:38:10.767336 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.767287    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m9qwf\""
Oct 30 18:38:10.767336 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.767298    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:38:10.768427 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/594ac1de-c6f2-4031-807e-08509c8a162d-serviceca\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:38:10.768496 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db2ck\" (UniqueName: \"kubernetes.io/projected/594ac1de-c6f2-4031-807e-08509c8a162d-kube-api-access-db2ck\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:38:10.768496 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r8lg\" (UniqueName: \"kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg\") pod \"network-check-target-27m75\" (UID: \"761ddebb-9140-4814-8145-3bb12ce8a148\") " pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:10.768595 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj64r\" (UniqueName: \"kubernetes.io/projected/45c93bbd-c046-4a50-83f2-ae8de8b64104-kube-api-access-gj64r\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:38:10.768595 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-socket-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.768595 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1395368e-c046-4067-809a-6310f8d3c93d-iptables-alerter-script\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:38:10.768595 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-dbus\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:10.768713 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-systemd-units\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.768713 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-run-netns\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.768713 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-etc-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.768713 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-etc-kubernetes\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.768821 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crdr9\" (UniqueName: \"kubernetes.io/projected/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-kube-api-access-crdr9\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:10.768821 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-lib-modules\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.768821 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-run-ovn-kubernetes\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.768821 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-system-cni-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.768924 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-cnibin\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.768924 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-netns\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.768924 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-modprobe-d\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.768924 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysctl-conf\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.769051 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/028108f5-f006-42d0-9cfd-56f9b55e18ff-tmp\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.769051 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2r6v\" (UniqueName: \"kubernetes.io/projected/028108f5-f006-42d0-9cfd-56f9b55e18ff-kube-api-access-d2r6v\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.769051 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.768962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-slash\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.769051 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.769051 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-k8s-cni-cncf-io\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.769243 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-systemd\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.769243 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769093    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:38:10.769243 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c93bbd-c046-4a50-83f2-ae8de8b64104-tmp-dir\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:38:10.769243 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-ovnkube-script-lib\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.769243 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.769243 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-cni-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.769243 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-socket-dir-parent\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.769243 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-conf-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-etc-selinux\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-run\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-node-log\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769334    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-ovnkube-config\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjnqm\" (UniqueName: \"kubernetes.io/projected/f95bb75f-1760-4132-bf36-20c2027b3059-kube-api-access-qjnqm\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-kubelet-config\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-var-lib-kubelet\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-tuned\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-env-overrides\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cnibin\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-sys-fs\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769489    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769493    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgcm\" (UniqueName: \"kubernetes.io/projected/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-kube-api-access-7vgcm\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.769554 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysconfig\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-kubernetes\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f95bb75f-1760-4132-bf36-20c2027b3059-multus-daemon-config\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769610    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q8mbb\""
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-registration-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-cni-netd\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-cni-multus\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-device-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d70a762-bdea-4394-8a95-5e5f0fb4042c-konnectivity-ca\") pod \"konnectivity-agent-wwvr8\" (UID: \"4d70a762-bdea-4394-8a95-5e5f0fb4042c\") " pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysctl-d\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-host\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f478498d-4adf-441d-9dc6-1988d7c9e050-ovn-node-metrics-cert\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-system-cni-dir\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-hostroot\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-sys\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.770113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/45c93bbd-c046-4a50-83f2-ae8de8b64104-hosts-file\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.769939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-862sw\" (UniqueName: \"kubernetes.io/projected/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-kube-api-access-862sw\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-multus-certs\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-kubelet\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-systemd\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-var-lib-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-ovn\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-cni-bin\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-kubelet\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzx9l\" (UniqueName: \"kubernetes.io/projected/f478498d-4adf-441d-9dc6-1988d7c9e050-kube-api-access-jzx9l\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f95bb75f-1760-4132-bf36-20c2027b3059-cni-binary-copy\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d70a762-bdea-4394-8a95-5e5f0fb4042c-agent-certs\") pod \"konnectivity-agent-wwvr8\" (UID: \"4d70a762-bdea-4394-8a95-5e5f0fb4042c\") " pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1395368e-c046-4067-809a-6310f8d3c93d-host-slash\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/594ac1de-c6f2-4031-807e-08509c8a162d-host\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:38:10.770738 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-log-socket\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.771182 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:10.771182 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-cni-bin\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.771182 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.771182 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbtd6\" (UniqueName: \"kubernetes.io/projected/1395368e-c046-4067-809a-6310f8d3c93d-kube-api-access-fbtd6\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:38:10.771182 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.770549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:10.813234 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.813193    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xlrq8\""
Oct 30 18:38:10.813234 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.813236    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:38:10.813370 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.813256    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:38:10.813370 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.813299    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:38:10.813370 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.813347    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s4fk8\""
Oct 30 18:38:10.813463 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.813370    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:38:10.813500 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.813471    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h468w\""
Oct 30 18:38:10.813500 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.813483    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:38:10.813744 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.813729    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:38:10.813804 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.813771    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:38:10.813944 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.813913    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:38:10.813944 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.813921    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-476j6\""
Oct 30 18:38:10.814079 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.813926    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:38:10.814931 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.814385    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:38:10.814931 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.814637    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:38:10.814931 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.814742    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:38:10.864081 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.864047    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:38:10.864205 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.864119    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-thmgz"
Oct 30 18:38:10.871499 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.871571 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzx9l\" (UniqueName: \"kubernetes.io/projected/f478498d-4adf-441d-9dc6-1988d7c9e050-kube-api-access-jzx9l\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.871571 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f95bb75f-1760-4132-bf36-20c2027b3059-cni-binary-copy\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.871571 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d70a762-bdea-4394-8a95-5e5f0fb4042c-agent-certs\") pod \"konnectivity-agent-wwvr8\" (UID: \"4d70a762-bdea-4394-8a95-5e5f0fb4042c\") " pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:38:10.871716 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1395368e-c046-4067-809a-6310f8d3c93d-host-slash\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:38:10.871771 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/594ac1de-c6f2-4031-807e-08509c8a162d-host\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:38:10.871830 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-log-socket\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.871830 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:10.871830 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871812    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1395368e-c046-4067-809a-6310f8d3c93d-host-slash\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:38:10.871830 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-cni-bin\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.872014 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871838    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:38:10.872014 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.872014 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871841    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/594ac1de-c6f2-4031-807e-08509c8a162d-host\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:38:10.872014 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbtd6\" (UniqueName: \"kubernetes.io/projected/1395368e-c046-4067-809a-6310f8d3c93d-kube-api-access-fbtd6\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:38:10.872014 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871857    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-log-socket\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.872014 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871900    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-cni-bin\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.872014 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:10.871933    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:10.872014 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871965    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.872014 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.871913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:10.872014 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:10.871997    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:10.872016    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs podName:e8659e3f-b750-4d38-aa0b-7cc7f883f9b4 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:11.371993779 +0000 UTC m=+2.283269242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs") pod "network-metrics-daemon-676kr" (UID: "e8659e3f-b750-4d38-aa0b-7cc7f883f9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872095    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f95bb75f-1760-4132-bf36-20c2027b3059-cni-binary-copy\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/594ac1de-c6f2-4031-807e-08509c8a162d-serviceca\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872111    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db2ck\" (UniqueName: \"kubernetes.io/projected/594ac1de-c6f2-4031-807e-08509c8a162d-kube-api-access-db2ck\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:10.872177    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret podName:ab2168bb-3bf1-48ed-993d-4d0a014f00ae nodeName:}" failed. No retries permitted until 2026-10-30 18:38:11.372158313 +0000 UTC m=+2.283433756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret") pod "global-pull-secret-syncer-7h2cm" (UID: "ab2168bb-3bf1-48ed-993d-4d0a014f00ae") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8lg\" (UniqueName: \"kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg\") pod \"network-check-target-27m75\" (UID: \"761ddebb-9140-4814-8145-3bb12ce8a148\") " pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj64r\" (UniqueName: \"kubernetes.io/projected/45c93bbd-c046-4a50-83f2-ae8de8b64104-kube-api-access-gj64r\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-socket-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1395368e-c046-4067-809a-6310f8d3c93d-iptables-alerter-script\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-dbus\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-systemd-units\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-run-netns\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-etc-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.872550 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-etc-kubernetes\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crdr9\" (UniqueName: \"kubernetes.io/projected/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-kube-api-access-crdr9\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-lib-modules\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-run-ovn-kubernetes\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-system-cni-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-cnibin\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872467    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-dbus\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872477    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-systemd-units\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872522    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-run-ovn-kubernetes\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872552    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/594ac1de-c6f2-4031-807e-08509c8a162d-serviceca\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872558    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-system-cni-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872582    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-cnibin\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872420    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-etc-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872604    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-etc-kubernetes\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872637    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-lib-modules\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-netns\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872669    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-netns\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872677    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-socket-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.873259 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872711    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-run-netns\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-modprobe-d\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysctl-conf\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/028108f5-f006-42d0-9cfd-56f9b55e18ff-tmp\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2r6v\" (UniqueName: \"kubernetes.io/projected/028108f5-f006-42d0-9cfd-56f9b55e18ff-kube-api-access-d2r6v\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872774    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-modprobe-d\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-slash\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872905    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysctl-conf\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872913    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-slash\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-k8s-cni-cncf-io\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-systemd\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c93bbd-c046-4a50-83f2-ae8de8b64104-tmp-dir\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-ovnkube-script-lib\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873003    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.872993    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1395368e-c046-4067-809a-6310f8d3c93d-iptables-alerter-script\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.874053 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873053    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-systemd\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-cni-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-socket-dir-parent\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-conf-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-etc-selinux\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-run\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-node-log\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-ovnkube-config\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873247    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-socket-dir-parent\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjnqm\" (UniqueName: \"kubernetes.io/projected/f95bb75f-1760-4132-bf36-20c2027b3059-kube-api-access-qjnqm\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-kubelet-config\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873306    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c93bbd-c046-4a50-83f2-ae8de8b64104-tmp-dir\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-var-lib-kubelet\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873344    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-etc-selinux\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-tuned\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-env-overrides\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873403    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-kubelet-config\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873403    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-run\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.874806 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cnibin\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873455    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-var-lib-kubelet\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-sys-fs\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873535    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-k8s-cni-cncf-io\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873194    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-cni-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873565    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-conf-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873592    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-node-log\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873567    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-ovnkube-script-lib\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgcm\" (UniqueName: \"kubernetes.io/projected/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-kube-api-access-7vgcm\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873643    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-sys-fs\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873646    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cnibin\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysconfig\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-kubernetes\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f95bb75f-1760-4132-bf36-20c2027b3059-multus-daemon-config\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-registration-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873520    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.875516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873798    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-cni-netd\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873847    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-cni-netd\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-cni-multus\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873888    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-kubernetes\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873915    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysconfig\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-device-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873945    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-env-overrides\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873955    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-registration-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d70a762-bdea-4394-8a95-5e5f0fb4042c-konnectivity-ca\") pod \"konnectivity-agent-wwvr8\" (UID: \"4d70a762-bdea-4394-8a95-5e5f0fb4042c\") " pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873974    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-ovnkube-config\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.873997    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-cni-multus\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874023    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-device-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysctl-d\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-host\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f478498d-4adf-441d-9dc6-1988d7c9e050-ovn-node-metrics-cert\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.876192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-system-cni-dir\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-hostroot\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-sys\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874314    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-host\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/45c93bbd-c046-4a50-83f2-ae8de8b64104-hosts-file\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874389    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/45c93bbd-c046-4a50-83f2-ae8de8b64104-hosts-file\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874421    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874436    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f95bb75f-1760-4132-bf36-20c2027b3059-multus-daemon-config\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874448    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysctl-d\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874480    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-hostroot\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874488    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-system-cni-dir\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862sw\" (UniqueName: \"kubernetes.io/projected/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-kube-api-access-862sw\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874528    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-sys\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-multus-certs\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-kubelet\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874565    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.876639 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-systemd\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-var-lib-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-kubelet\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-ovn\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874628    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-multus-certs\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-cni-bin\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874654    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-var-lib-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-kubelet\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874688    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-systemd\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874700    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-ovn\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874785    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-kubelet\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.874800    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-cni-bin\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.875274    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d70a762-bdea-4394-8a95-5e5f0fb4042c-konnectivity-ca\") pod \"konnectivity-agent-wwvr8\" (UID: \"4d70a762-bdea-4394-8a95-5e5f0fb4042c\") " pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.875877    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d70a762-bdea-4394-8a95-5e5f0fb4042c-agent-certs\") pod \"konnectivity-agent-wwvr8\" (UID: \"4d70a762-bdea-4394-8a95-5e5f0fb4042c\") " pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.875999    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/028108f5-f006-42d0-9cfd-56f9b55e18ff-tmp\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.877172 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.876001    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-tuned\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.877595 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.877191    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f478498d-4adf-441d-9dc6-1988d7c9e050-ovn-node-metrics-cert\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.926388 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:10.926354    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:10.926388 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:10.926378    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:10.926388 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:10.926391    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4r8lg for pod openshift-network-diagnostics/network-check-target-27m75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:10.926594 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:10.926453    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg podName:761ddebb-9140-4814-8145-3bb12ce8a148 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:11.426430999 +0000 UTC m=+2.337706442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4r8lg" (UniqueName: "kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg") pod "network-check-target-27m75" (UID: "761ddebb-9140-4814-8145-3bb12ce8a148") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:10.928728 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.928696    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbtd6\" (UniqueName: \"kubernetes.io/projected/1395368e-c046-4067-809a-6310f8d3c93d-kube-api-access-fbtd6\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:38:10.929122 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.929099    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzx9l\" (UniqueName: \"kubernetes.io/projected/f478498d-4adf-441d-9dc6-1988d7c9e050-kube-api-access-jzx9l\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:10.929240 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.929128    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db2ck\" (UniqueName: \"kubernetes.io/projected/594ac1de-c6f2-4031-807e-08509c8a162d-kube-api-access-db2ck\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:38:10.929888 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.929870    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgcm\" (UniqueName: \"kubernetes.io/projected/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-kube-api-access-7vgcm\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:10.930014 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.929907    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj64r\" (UniqueName: \"kubernetes.io/projected/45c93bbd-c046-4a50-83f2-ae8de8b64104-kube-api-access-gj64r\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:38:10.930100 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.929949    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crdr9\" (UniqueName: \"kubernetes.io/projected/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-kube-api-access-crdr9\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:10.930165 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.930151    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2r6v\" (UniqueName: \"kubernetes.io/projected/028108f5-f006-42d0-9cfd-56f9b55e18ff-kube-api-access-d2r6v\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:10.930680 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.930637    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-862sw\" (UniqueName: \"kubernetes.io/projected/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-kube-api-access-862sw\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:10.930958 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:10.930944    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjnqm\" (UniqueName: \"kubernetes.io/projected/f95bb75f-1760-4132-bf36-20c2027b3059-kube-api-access-qjnqm\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:38:11.049490 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.049461    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:38:11.059841 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.059812    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pkq65"
Oct 30 18:38:11.067052 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:11.067018    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349d28f3fc367c9ffa500d3d4a12aae5.slice/crio-3309f32908643a7ebc30a95f6d73c02dfdfe2734a33d056980f0e964cdf091b4 WatchSource:0}: Error finding container 3309f32908643a7ebc30a95f6d73c02dfdfe2734a33d056980f0e964cdf091b4: Status 404 returned error can't find the container with id 3309f32908643a7ebc30a95f6d73c02dfdfe2734a33d056980f0e964cdf091b4
Oct 30 18:38:11.067980 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:11.067957    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e259e9_ce5d_458d_8b2f_7a5e4448e587.slice/crio-2de8db5c3bbc09064ad9052704fc421a8149e9c5a794d5148d74941f4ec1ea47 WatchSource:0}: Error finding container 2de8db5c3bbc09064ad9052704fc421a8149e9c5a794d5148d74941f4ec1ea47: Status 404 returned error can't find the container with id 2de8db5c3bbc09064ad9052704fc421a8149e9c5a794d5148d74941f4ec1ea47
Oct 30 18:38:11.068793 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:11.068776    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a238862e2cf33c9dc387d53f60c786c.slice/crio-8f0941b9270fcdcf0be5202685ed7f97e1a19a72fb544148ab8ce523fad9437f WatchSource:0}: Error finding container 8f0941b9270fcdcf0be5202685ed7f97e1a19a72fb544148ab8ce523fad9437f: Status 404 returned error can't find the container with id 8f0941b9270fcdcf0be5202685ed7f97e1a19a72fb544148ab8ce523fad9437f
Oct 30 18:38:11.071785 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:11.071767    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95bb75f_1760_4132_bf36_20c2027b3059.slice/crio-2e5aca942a63fe3102bf2f2c2b6bad1e47166e81b0f9547008ba4a8e6ad32ac5 WatchSource:0}: Error finding container 2e5aca942a63fe3102bf2f2c2b6bad1e47166e81b0f9547008ba4a8e6ad32ac5: Status 404 returned error can't find the container with id 2e5aca942a63fe3102bf2f2c2b6bad1e47166e81b0f9547008ba4a8e6ad32ac5
Oct 30 18:38:11.073693 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.073676    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:38:11.081233 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.081207    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:38:11.090118 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:11.090097    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594ac1de_c6f2_4031_807e_08509c8a162d.slice/crio-333698a8e60fe161576bb7e574f396a93f76b1f7440a50376c71ad6df33db7e3 WatchSource:0}: Error finding container 333698a8e60fe161576bb7e574f396a93f76b1f7440a50376c71ad6df33db7e3: Status 404 returned error can't find the container with id 333698a8e60fe161576bb7e574f396a93f76b1f7440a50376c71ad6df33db7e3
Oct 30 18:38:11.095222 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.095206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:38:11.102858 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:11.102832    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028108f5_f006_42d0_9cfd_56f9b55e18ff.slice/crio-d06e0db0bdca33c4f0affa5d5ee777db357a6796c563c014733f5b7592489b30 WatchSource:0}: Error finding container d06e0db0bdca33c4f0affa5d5ee777db357a6796c563c014733f5b7592489b30: Status 404 returned error can't find the container with id d06e0db0bdca33c4f0affa5d5ee777db357a6796c563c014733f5b7592489b30
Oct 30 18:38:11.108920 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.108902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:38:11.116242 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:11.116219    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5a6b4e_719e_4978_afa3_8d179bbcbdd2.slice/crio-ce815c96b5f3d14b0ca6eaea2f90f65a441db6cf7634b899d735dce29ab8a452 WatchSource:0}: Error finding container ce815c96b5f3d14b0ca6eaea2f90f65a441db6cf7634b899d735dce29ab8a452: Status 404 returned error can't find the container with id ce815c96b5f3d14b0ca6eaea2f90f65a441db6cf7634b899d735dce29ab8a452
Oct 30 18:38:11.123192 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.123171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:38:11.131507 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:11.131489    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d70a762_bdea_4394_8a95_5e5f0fb4042c.slice/crio-30023e8fbff341b10235f3e2091641296d825c74991ff9edf6568ad8934352b7 WatchSource:0}: Error finding container 30023e8fbff341b10235f3e2091641296d825c74991ff9edf6568ad8934352b7: Status 404 returned error can't find the container with id 30023e8fbff341b10235f3e2091641296d825c74991ff9edf6568ad8934352b7
Oct 30 18:38:11.142022 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.142002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:38:11.151366 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:11.151339    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1395368e_c046_4067_809a_6310f8d3c93d.slice/crio-30ea2ad9a892d2d7ff1080af124641c146d08ae1cc4cba0ecad051080b21e84c WatchSource:0}: Error finding container 30ea2ad9a892d2d7ff1080af124641c146d08ae1cc4cba0ecad051080b21e84c: Status 404 returned error can't find the container with id 30ea2ad9a892d2d7ff1080af124641c146d08ae1cc4cba0ecad051080b21e84c
Oct 30 18:38:11.156352 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.156336    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:38:11.164540 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:11.164518    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c93bbd_c046_4a50_83f2_ae8de8b64104.slice/crio-3ab2b76243302770f606dbe9948645a4929f6281387f5e93e8ea3742609236dc WatchSource:0}: Error finding container 3ab2b76243302770f606dbe9948645a4929f6281387f5e93e8ea3742609236dc: Status 404 returned error can't find the container with id 3ab2b76243302770f606dbe9948645a4929f6281387f5e93e8ea3742609236dc
Oct 30 18:38:11.165812 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.165799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:11.173877 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:11.173857    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf478498d_4adf_441d_9dc6_1988d7c9e050.slice/crio-a557ca466124a05b24eac067708fabba82e0fb74c3cbed4d5737a4d43e9edf07 WatchSource:0}: Error finding container a557ca466124a05b24eac067708fabba82e0fb74c3cbed4d5737a4d43e9edf07: Status 404 returned error can't find the container with id a557ca466124a05b24eac067708fabba82e0fb74c3cbed4d5737a4d43e9edf07
Oct 30 18:38:11.175803 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:11.175773    2578 kuberuntime_manager.go:1707] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-54qsj_openshift-ovn-kubernetes(f478498d-4adf-441d-9dc6-1988d7c9e050): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Oct 30 18:38:11.177674 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:11.177658    2578 kuberuntime_manager.go:1707] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-54qsj_openshift-ovn-kubernetes(f478498d-4adf-441d-9dc6-1988d7c9e050): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Oct 30 18:38:11.181890 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:11.181846    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj" podUID="f478498d-4adf-441d-9dc6-1988d7c9e050"
Oct 30 18:38:11.330504 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.330476    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:11.349624 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.349599    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:11.377902 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.377873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:11.377902 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.377902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:11.378101 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:11.377987    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:11.378101 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:11.378006    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:11.378101 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:11.378034    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret podName:ab2168bb-3bf1-48ed-993d-4d0a014f00ae nodeName:}" failed. No retries permitted until 2026-10-30 18:38:12.378023611 +0000 UTC m=+3.289299053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret") pod "global-pull-secret-syncer-7h2cm" (UID: "ab2168bb-3bf1-48ed-993d-4d0a014f00ae") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:11.378101 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:11.378053    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs podName:e8659e3f-b750-4d38-aa0b-7cc7f883f9b4 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:12.378041761 +0000 UTC m=+3.289317206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs") pod "network-metrics-daemon-676kr" (UID: "e8659e3f-b750-4d38-aa0b-7cc7f883f9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:11.445490 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.445412    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:11.478577 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.478551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8lg\" (UniqueName: \"kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg\") pod \"network-check-target-27m75\" (UID: \"761ddebb-9140-4814-8145-3bb12ce8a148\") " pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:11.478696 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:11.478668    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:11.478696 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:11.478682    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:11.478696 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:11.478690    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4r8lg for pod openshift-network-diagnostics/network-check-target-27m75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:11.478790 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:11.478730    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg podName:761ddebb-9140-4814-8145-3bb12ce8a148 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:12.47871954 +0000 UTC m=+3.389994982 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4r8lg" (UniqueName: "kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg") pod "network-check-target-27m75" (UID: "761ddebb-9140-4814-8145-3bb12ce8a148") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:11.615277 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.615227    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn" event={"ID":"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2","Type":"ContainerStarted","Data":"ce815c96b5f3d14b0ca6eaea2f90f65a441db6cf7634b899d735dce29ab8a452"}
Oct 30 18:38:11.616175 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.616156    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8pwf7" event={"ID":"594ac1de-c6f2-4031-807e-08509c8a162d","Type":"ContainerStarted","Data":"333698a8e60fe161576bb7e574f396a93f76b1f7440a50376c71ad6df33db7e3"}
Oct 30 18:38:11.617090 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.617066    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pkq65" event={"ID":"f95bb75f-1760-4132-bf36-20c2027b3059","Type":"ContainerStarted","Data":"2e5aca942a63fe3102bf2f2c2b6bad1e47166e81b0f9547008ba4a8e6ad32ac5"}
Oct 30 18:38:11.617959 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.617938    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal" event={"ID":"5a238862e2cf33c9dc387d53f60c786c","Type":"ContainerStarted","Data":"8f0941b9270fcdcf0be5202685ed7f97e1a19a72fb544148ab8ce523fad9437f"}
Oct 30 18:38:11.618808 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.618782    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal" event={"ID":"349d28f3fc367c9ffa500d3d4a12aae5","Type":"ContainerStarted","Data":"3309f32908643a7ebc30a95f6d73c02dfdfe2734a33d056980f0e964cdf091b4"}
Oct 30 18:38:11.619792 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.619777    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj" event={"ID":"f478498d-4adf-441d-9dc6-1988d7c9e050","Type":"ContainerStarted","Data":"a557ca466124a05b24eac067708fabba82e0fb74c3cbed4d5737a4d43e9edf07"}
Oct 30 18:38:11.620925 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.620910    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ctrbm" event={"ID":"1395368e-c046-4067-809a-6310f8d3c93d","Type":"ContainerStarted","Data":"30ea2ad9a892d2d7ff1080af124641c146d08ae1cc4cba0ecad051080b21e84c"}
Oct 30 18:38:11.621940 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.621924    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wwvr8" event={"ID":"4d70a762-bdea-4394-8a95-5e5f0fb4042c","Type":"ContainerStarted","Data":"30023e8fbff341b10235f3e2091641296d825c74991ff9edf6568ad8934352b7"}
Oct 30 18:38:11.622992 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.622974    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-plh42" event={"ID":"028108f5-f006-42d0-9cfd-56f9b55e18ff","Type":"ContainerStarted","Data":"d06e0db0bdca33c4f0affa5d5ee777db357a6796c563c014733f5b7592489b30"}
Oct 30 18:38:11.624093 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.624075    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bjz2" event={"ID":"b2e259e9-ce5d-458d-8b2f-7a5e4448e587","Type":"ContainerStarted","Data":"2de8db5c3bbc09064ad9052704fc421a8149e9c5a794d5148d74941f4ec1ea47"}
Oct 30 18:38:11.626904 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.626886    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7mfxz" event={"ID":"45c93bbd-c046-4a50-83f2-ae8de8b64104","Type":"ContainerStarted","Data":"3ab2b76243302770f606dbe9948645a4929f6281387f5e93e8ea3742609236dc"}
Oct 30 18:38:11.629415 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:11.629358    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj" podUID="f478498d-4adf-441d-9dc6-1988d7c9e050"
Oct 30 18:38:11.866020 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.865909    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:33:10 +0000 UTC" deadline="2028-06-29 22:25:39.09162032 +0000 UTC"
Oct 30 18:38:11.866020 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:11.865967    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14595h47m27.225657374s"
Oct 30 18:38:12.386037 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:12.385614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:12.386037 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:12.385668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:12.386037 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:12.385837    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:12.386037 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:12.385903    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret podName:ab2168bb-3bf1-48ed-993d-4d0a014f00ae nodeName:}" failed. No retries permitted until 2026-10-30 18:38:14.385885081 +0000 UTC m=+5.297160547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret") pod "global-pull-secret-syncer-7h2cm" (UID: "ab2168bb-3bf1-48ed-993d-4d0a014f00ae") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:12.386037 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:12.385971    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:12.386037 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:12.386001    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs podName:e8659e3f-b750-4d38-aa0b-7cc7f883f9b4 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:14.385990974 +0000 UTC m=+5.297266417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs") pod "network-metrics-daemon-676kr" (UID: "e8659e3f-b750-4d38-aa0b-7cc7f883f9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:12.486490 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:12.486447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8lg\" (UniqueName: \"kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg\") pod \"network-check-target-27m75\" (UID: \"761ddebb-9140-4814-8145-3bb12ce8a148\") " pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:12.486861 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:12.486645    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:12.486861 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:12.486663    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:12.486861 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:12.486675    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4r8lg for pod openshift-network-diagnostics/network-check-target-27m75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:12.486861 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:12.486736    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg podName:761ddebb-9140-4814-8145-3bb12ce8a148 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:14.486714438 +0000 UTC m=+5.397989895 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4r8lg" (UniqueName: "kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg") pod "network-check-target-27m75" (UID: "761ddebb-9140-4814-8145-3bb12ce8a148") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:12.611888 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:12.611855    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:12.612042 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:12.611997    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:12.612089 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:12.612077    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:12.612192 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:12.612172    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:12.612243 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:12.612236    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:12.612296 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:12.612283    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:12.866234 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:12.866109    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:33:10 +0000 UTC" deadline="2028-07-26 22:35:23.206391378 +0000 UTC"
Oct 30 18:38:12.866234 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:12.866183    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15243h57m10.340211932s"
Oct 30 18:38:14.402920 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:14.402699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:14.402920 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:14.402749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:14.402920 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:14.402898    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:14.403513 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:14.402961    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret podName:ab2168bb-3bf1-48ed-993d-4d0a014f00ae nodeName:}" failed. No retries permitted until 2026-10-30 18:38:18.402943511 +0000 UTC m=+9.314218955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret") pod "global-pull-secret-syncer-7h2cm" (UID: "ab2168bb-3bf1-48ed-993d-4d0a014f00ae") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:14.403513 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:14.403036    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:14.403513 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:14.403072    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs podName:e8659e3f-b750-4d38-aa0b-7cc7f883f9b4 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:18.403059326 +0000 UTC m=+9.314334783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs") pod "network-metrics-daemon-676kr" (UID: "e8659e3f-b750-4d38-aa0b-7cc7f883f9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:14.503651 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:14.503611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8lg\" (UniqueName: \"kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg\") pod \"network-check-target-27m75\" (UID: \"761ddebb-9140-4814-8145-3bb12ce8a148\") " pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:14.503821 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:14.503802    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:14.503891 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:14.503828    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:14.503891 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:14.503842    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4r8lg for pod openshift-network-diagnostics/network-check-target-27m75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:14.503989 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:14.503901    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg podName:761ddebb-9140-4814-8145-3bb12ce8a148 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:18.503882992 +0000 UTC m=+9.415158455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4r8lg" (UniqueName: "kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg") pod "network-check-target-27m75" (UID: "761ddebb-9140-4814-8145-3bb12ce8a148") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:14.612451 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:14.612009    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:14.612451 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:14.612404    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:14.612697 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:14.612402    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:14.612697 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:14.612514    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:14.612697 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:14.612526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:14.612697 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:14.612595    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:14.642250 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:14.642203    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" containerID="10c6d83d51c796d56975d832d24b7e46c8e17f4228223d295c501ad48a887746" exitCode=0
Oct 30 18:38:14.642411 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:14.642260    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bjz2" event={"ID":"b2e259e9-ce5d-458d-8b2f-7a5e4448e587","Type":"ContainerDied","Data":"10c6d83d51c796d56975d832d24b7e46c8e17f4228223d295c501ad48a887746"}
Oct 30 18:38:15.665212 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:15.646526    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wwvr8" event={"ID":"4d70a762-bdea-4394-8a95-5e5f0fb4042c","Type":"ContainerStarted","Data":"446204f86e9366a9ee958b3eefd7c1042254510af4dddcd4e0d8441f9f83d427"}
Oct 30 18:38:15.665212 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:15.648218    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal" event={"ID":"5a238862e2cf33c9dc387d53f60c786c","Type":"ContainerStarted","Data":"73ed0227f00f949e0a46e74ccfa42589f3069c5c9a600863dbde210f1d9b02f8"}
Oct 30 18:38:15.665739 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:15.665638    2578 generic.go:356] "Generic (PLEG): container finished" podID="349d28f3fc367c9ffa500d3d4a12aae5" containerID="5ec01d92ca7eb5ff7183b4a6d17b942ca0d98ca6333f662dbdac96f4efe207c2" exitCode=0
Oct 30 18:38:15.665739 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:15.665673    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal" event={"ID":"349d28f3fc367c9ffa500d3d4a12aae5","Type":"ContainerDied","Data":"5ec01d92ca7eb5ff7183b4a6d17b942ca0d98ca6333f662dbdac96f4efe207c2"}
Oct 30 18:38:15.879902 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:15.871821    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wwvr8" podStartSLOduration=2.6442729209999998 podStartE2EDuration="6.871805151s" totalImagesPullingTime="4.22753223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:15.759824646 +0000 UTC m=+6.671100112" watchObservedRunningTime="2026-10-30 18:38:15.871805151 +0000 UTC m=+6.783080615"
Oct 30 18:38:16.613107 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:16.612218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:16.613107 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:16.612394    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:16.613107 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:16.612792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:16.613107 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:16.612861    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:16.613107 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:16.612906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:16.613107 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:16.612982    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:16.670525 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:16.670485    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal" event={"ID":"349d28f3fc367c9ffa500d3d4a12aae5","Type":"ContainerStarted","Data":"7da45b4d8dd8a7950fce51a3bf0d66cc0b6ec01a425049e8fb55ca2915f34a57"}
Oct 30 18:38:16.677336 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:16.677305    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8pwf7" event={"ID":"594ac1de-c6f2-4031-807e-08509c8a162d","Type":"ContainerStarted","Data":"adf1a99e962427fba43162532ca69ea8d85e84df7a13f0fbcf18f82b022fa37e"}
Oct 30 18:38:16.781719 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:16.781659    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal" podStartSLOduration=7.781632149 podStartE2EDuration="7.781632149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:15.986469528 +0000 UTC m=+6.897744988" watchObservedRunningTime="2026-10-30 18:38:16.781632149 +0000 UTC m=+7.692907618"
Oct 30 18:38:16.893669 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:16.893106    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal" podStartSLOduration=7.893091654 podStartE2EDuration="7.893091654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:16.782240087 +0000 UTC m=+7.693515553" watchObservedRunningTime="2026-10-30 18:38:16.893091654 +0000 UTC m=+7.804367119"
Oct 30 18:38:16.918285 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:16.918246    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:38:17.681925 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:17.681413    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn" event={"ID":"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2","Type":"ContainerStarted","Data":"ce0065419be8ff514fb3f0b0d2754ce03102242762d10c13b390c3921901c1de"}
Oct 30 18:38:18.441928 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:18.441618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:18.441928 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:18.441653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:18.441928 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:18.441769    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:18.441928 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:18.441816    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret podName:ab2168bb-3bf1-48ed-993d-4d0a014f00ae nodeName:}" failed. No retries permitted until 2026-10-30 18:38:26.441798882 +0000 UTC m=+17.353074341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret") pod "global-pull-secret-syncer-7h2cm" (UID: "ab2168bb-3bf1-48ed-993d-4d0a014f00ae") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:18.441928 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:18.441861    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:18.441928 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:18.441883    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs podName:e8659e3f-b750-4d38-aa0b-7cc7f883f9b4 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:26.441876302 +0000 UTC m=+17.353151745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs") pod "network-metrics-daemon-676kr" (UID: "e8659e3f-b750-4d38-aa0b-7cc7f883f9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:18.543026 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:18.542990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8lg\" (UniqueName: \"kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg\") pod \"network-check-target-27m75\" (UID: \"761ddebb-9140-4814-8145-3bb12ce8a148\") " pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:18.543209 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:18.543171    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:18.543209 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:18.543186    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:18.543209 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:18.543195    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4r8lg for pod openshift-network-diagnostics/network-check-target-27m75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:18.543370 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:18.543237    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg podName:761ddebb-9140-4814-8145-3bb12ce8a148 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:26.543222539 +0000 UTC m=+17.454497982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4r8lg" (UniqueName: "kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg") pod "network-check-target-27m75" (UID: "761ddebb-9140-4814-8145-3bb12ce8a148") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:18.612413 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:18.611901    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:18.612413 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:18.612007    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:18.612413 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:18.612060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:18.612413 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:18.612105    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:18.612413 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:18.612151    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:18.612413 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:18.612190    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:18.688690 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:18.688006    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pkq65" event={"ID":"f95bb75f-1760-4132-bf36-20c2027b3059","Type":"ContainerStarted","Data":"1d48c35123104d58d8f4a52946f18ecd696a5913560e178c47caf076cc21d927"}
Oct 30 18:38:18.809797 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:18.809759    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:38:18.835699 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:18.835652    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pkq65" podStartSLOduration=2.5763265459999998 podStartE2EDuration="9.835639198s" totalImagesPullingTime="7.259312652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:18.835265627 +0000 UTC m=+9.746541093" watchObservedRunningTime="2026-10-30 18:38:18.835639198 +0000 UTC m=+9.746914662"
Oct 30 18:38:18.835962 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:18.835937    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8pwf7" podStartSLOduration=4.744999723 podStartE2EDuration="9.835931046s" totalImagesPullingTime="5.090931323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:16.894314459 +0000 UTC m=+7.805589915" watchObservedRunningTime="2026-10-30 18:38:18.835931046 +0000 UTC m=+9.747206510"
Oct 30 18:38:19.536163 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:19.536058    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:38:19.536840 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:19.536819    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:38:19.562363 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:19.562253    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:38:18.809775575Z","UUID":"dd06e9f7-e0a0-4b1f-ae5e-d4744512d8ab","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:38:19.563693 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:19.563671    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:38:19.563814 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:19.563706    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:38:19.691790 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:19.691760    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" containerID="c7c7bba50f1532576ee01b575f7356ffa55201874d30d02c3259210f96a27206" exitCode=0
Oct 30 18:38:19.692215 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:19.691846    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bjz2" event={"ID":"b2e259e9-ce5d-458d-8b2f-7a5e4448e587","Type":"ContainerDied","Data":"c7c7bba50f1532576ee01b575f7356ffa55201874d30d02c3259210f96a27206"}
Oct 30 18:38:19.693596 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:19.693567    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7mfxz" event={"ID":"45c93bbd-c046-4a50-83f2-ae8de8b64104","Type":"ContainerStarted","Data":"70469c9f753334ec71391fee0c7bc509a9e421d4713195f9bd1a4eb855e23267"}
Oct 30 18:38:19.695690 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:19.695666    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn" event={"ID":"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2","Type":"ContainerStarted","Data":"e08869f361339e1360a5ec99f9970069ead12d7815846b65435b3c1c746ac8cf"}
Oct 30 18:38:19.696397 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:19.696375    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:38:19.814240 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:19.814113    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7mfxz" podStartSLOduration=2.3112379499999998 podStartE2EDuration="9.814095742s" totalImagesPullingTime="7.502857792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:19.813709479 +0000 UTC m=+10.724984962" watchObservedRunningTime="2026-10-30 18:38:19.814095742 +0000 UTC m=+10.725371210"
Oct 30 18:38:20.611758 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:20.611513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:20.611896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:20.611524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:20.611896 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:20.611784    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:20.611896 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:20.611862    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:20.611896 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:20.611567    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:20.612056 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:20.611927    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:20.698846 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:20.698811    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ctrbm" event={"ID":"1395368e-c046-4067-809a-6310f8d3c93d","Type":"ContainerStarted","Data":"5daded6062f4524a216219ba108812b976da23e898c1b467422726787cf71a56"}
Oct 30 18:38:20.700404 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:20.700376    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-plh42" event={"ID":"028108f5-f006-42d0-9cfd-56f9b55e18ff","Type":"ContainerStarted","Data":"b96a766c59d7db8cd928260d53afa48a323eb1ce44a467c26459cc847d484ef6"}
Oct 30 18:38:20.702463 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:20.702435    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn" event={"ID":"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2","Type":"ContainerStarted","Data":"5cc332de6aea1ad1cbdb82916c94e6dc4ff3538c55b7d3d672dbf7e8046aca9e"}
Oct 30 18:38:20.810926 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:20.810847    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ctrbm" podStartSLOduration=4.6378881960000005 podStartE2EDuration="11.810833492s" totalImagesPullingTime="7.172945296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:20.810744627 +0000 UTC m=+11.722020091" watchObservedRunningTime="2026-10-30 18:38:20.810833492 +0000 UTC m=+11.722108957"
Oct 30 18:38:20.927415 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:20.925454    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn" podStartSLOduration=3.074727413 podStartE2EDuration="11.925436315s" totalImagesPullingTime="8.850708902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:20.925158016 +0000 UTC m=+11.836433478" watchObservedRunningTime="2026-10-30 18:38:20.925436315 +0000 UTC m=+11.836711780"
Oct 30 18:38:21.040009 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:21.039959    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-plh42" podStartSLOduration=2.992653408 podStartE2EDuration="12.039944613s" totalImagesPullingTime="9.047291205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:21.039923221 +0000 UTC m=+11.951198680" watchObservedRunningTime="2026-10-30 18:38:21.039944613 +0000 UTC m=+11.951220077"
Oct 30 18:38:21.706246 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:21.706210    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" containerID="1fc4788dddef4992b313771aff7147eb44bbea4601280a067bc1b3feb7ec010a" exitCode=0
Oct 30 18:38:21.706765 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:21.706310    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bjz2" event={"ID":"b2e259e9-ce5d-458d-8b2f-7a5e4448e587","Type":"ContainerDied","Data":"1fc4788dddef4992b313771aff7147eb44bbea4601280a067bc1b3feb7ec010a"}
Oct 30 18:38:22.571336 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:22.571306    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7mfxz_45c93bbd-c046-4a50-83f2-ae8de8b64104/dns-node-resolver/0.log"
Oct 30 18:38:22.612270 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:22.612243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:22.612270 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:22.612265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:22.612465 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:22.612408    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:22.612465 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:22.612447    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:22.612575 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:22.612556    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:22.612658 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:22.612640    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:23.179046 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:23.179023    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8pwf7_594ac1de-c6f2-4031-807e-08509c8a162d/node-ca/0.log"
Oct 30 18:38:23.713004 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:23.712971    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" containerID="4ede6998f73228457b56c4263df7e0f76a11652555e90546b3da65598b41b24f" exitCode=0
Oct 30 18:38:23.713214 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:23.713018    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bjz2" event={"ID":"b2e259e9-ce5d-458d-8b2f-7a5e4448e587","Type":"ContainerDied","Data":"4ede6998f73228457b56c4263df7e0f76a11652555e90546b3da65598b41b24f"}
Oct 30 18:38:24.611530 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:24.611490    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:24.611973 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:24.611598    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:24.611973 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:24.611511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:24.611973 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:24.611502    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:24.611973 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:24.611662    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:24.611973 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:24.611760    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:26.505580 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:26.505362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:26.506052 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:26.505621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:26.506052 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:26.505530    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:26.506052 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:26.505741    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs podName:e8659e3f-b750-4d38-aa0b-7cc7f883f9b4 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:42.505719327 +0000 UTC m=+33.416994781 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs") pod "network-metrics-daemon-676kr" (UID: "e8659e3f-b750-4d38-aa0b-7cc7f883f9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:26.506052 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:26.505750    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:26.506052 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:26.505804    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret podName:ab2168bb-3bf1-48ed-993d-4d0a014f00ae nodeName:}" failed. No retries permitted until 2026-10-30 18:38:42.505790391 +0000 UTC m=+33.417065834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret") pod "global-pull-secret-syncer-7h2cm" (UID: "ab2168bb-3bf1-48ed-993d-4d0a014f00ae") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:26.606093 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:26.606051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8lg\" (UniqueName: \"kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg\") pod \"network-check-target-27m75\" (UID: \"761ddebb-9140-4814-8145-3bb12ce8a148\") " pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:26.606303 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:26.606252    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:26.606303 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:26.606274    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:26.606303 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:26.606286    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4r8lg for pod openshift-network-diagnostics/network-check-target-27m75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:26.606453 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:26.606340    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg podName:761ddebb-9140-4814-8145-3bb12ce8a148 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:42.606323653 +0000 UTC m=+33.517599100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4r8lg" (UniqueName: "kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg") pod "network-check-target-27m75" (UID: "761ddebb-9140-4814-8145-3bb12ce8a148") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:26.612170 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:26.612132    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:26.612297 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:26.612157    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:26.612297 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:26.612236    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:26.612394 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:26.612328    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:26.612447 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:26.612404    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:26.612611 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:26.612591    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:27.725989 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:27.725925    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" containerID="6b8d8243e4acfc605aed8b66e020f742f9fc8b931c03a1b79d858b3b0a453da4" exitCode=0
Oct 30 18:38:27.726465 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:27.725990    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bjz2" event={"ID":"b2e259e9-ce5d-458d-8b2f-7a5e4448e587","Type":"ContainerDied","Data":"6b8d8243e4acfc605aed8b66e020f742f9fc8b931c03a1b79d858b3b0a453da4"}
Oct 30 18:38:28.612173 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:28.611910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:28.612173 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:28.611941    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:28.612173 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:28.612028    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:28.612173 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:28.612093    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:28.612482 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:28.612223    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:28.612482 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:28.612312    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:28.732187 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:28.732158    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" containerID="023b0b58425ce5b846445e1b4169ec231dacf047bba51dd8db786d011f2aed04" exitCode=0
Oct 30 18:38:28.732628 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:28.732209    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bjz2" event={"ID":"b2e259e9-ce5d-458d-8b2f-7a5e4448e587","Type":"ContainerDied","Data":"023b0b58425ce5b846445e1b4169ec231dacf047bba51dd8db786d011f2aed04"}
Oct 30 18:38:29.740395 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:29.740357    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bjz2" event={"ID":"b2e259e9-ce5d-458d-8b2f-7a5e4448e587","Type":"ContainerStarted","Data":"d478ce2f9cbd34243acc1529f510093f5800fbc5412ac3bb2b8556cc0e458996"}
Oct 30 18:38:29.858382 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:29.858318    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6bjz2" podStartSLOduration=8.26683848 podStartE2EDuration="20.858303848s" totalImagesPullingTime="12.591465368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:29.857608466 +0000 UTC m=+20.768883965" watchObservedRunningTime="2026-10-30 18:38:29.858303848 +0000 UTC m=+20.769579313"
Oct 30 18:38:30.611840 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:30.611801    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:30.611840 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:30.611829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:30.612084 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:30.611818    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:30.612084 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:30.611927    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:30.612084 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:30.612026    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:30.612194 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:30.612108    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:32.612073 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:32.612034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:32.612687 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:32.612033    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:32.612687 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:32.612207    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:32.612687 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:32.612256    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:32.612687 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:32.612033    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:32.612687 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:32.612330    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:34.611672 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:34.611623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:34.611672 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:34.611624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:34.612230 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:34.611790    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:34.612230 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:34.611791    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:34.612230 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:34.611914    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:34.612230 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:34.612005    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:36.612127 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:36.611939    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:36.612549 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:36.611967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:36.612549 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:36.612223    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:36.612549 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:36.611989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:36.612549 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:36.612288    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:36.612549 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:36.612383    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:36.755778 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:36.755747    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj" event={"ID":"f478498d-4adf-441d-9dc6-1988d7c9e050","Type":"ContainerStarted","Data":"c7fe0181b6cc030f8ac999f3b812d6e49287c486caf6067bf3b028813e69ea04"}
Oct 30 18:38:36.755932 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:36.755811    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj" event={"ID":"f478498d-4adf-441d-9dc6-1988d7c9e050","Type":"ContainerStarted","Data":"e8e6359d9764c5a2ca89ce05a96a1e7437f6ca38ae6cddbe4ce123da43801fc7"}
Oct 30 18:38:36.755932 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:36.755841    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj" event={"ID":"f478498d-4adf-441d-9dc6-1988d7c9e050","Type":"ContainerStarted","Data":"5ade8c11623cc588b2d88b193e32068d2ad7d263c601bc4b6a54a9e908a05bed"}
Oct 30 18:38:36.755932 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:36.755870    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj" event={"ID":"f478498d-4adf-441d-9dc6-1988d7c9e050","Type":"ContainerStarted","Data":"f185edd5bb8214fa3a847467d3ff0e026c54c34f2c71227b3eaed7090f1e9305"}
Oct 30 18:38:36.755932 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:36.755896    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj" event={"ID":"f478498d-4adf-441d-9dc6-1988d7c9e050","Type":"ContainerStarted","Data":"922bfe9c746dab3f0b5a61aad2fe3a5aa102b27461bb06c72294d00dd96adff5"}
Oct 30 18:38:36.755932 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:36.755921    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj" event={"ID":"f478498d-4adf-441d-9dc6-1988d7c9e050","Type":"ContainerStarted","Data":"2c7ae286f4ca6fda8e9ffe5155678484af2ef740d893fb1741459cba586fcf35"}
Oct 30 18:38:38.612330 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:38.612292    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:38.612834 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:38.612358    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:38.612834 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:38.612439    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:38.612834 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:38.612458    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:38.612834 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:38.612506    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:38.612834 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:38.612576    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:38.763498 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:38.763280    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj" event={"ID":"f478498d-4adf-441d-9dc6-1988d7c9e050","Type":"ContainerStarted","Data":"6fcd0fe4ccb9cdbc3e8bfc83973dada72a45bdd38340dc1d717c98343010b559"}
Oct 30 18:38:40.611922 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:40.611893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:40.612616 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:40.611980    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:40.612616 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:40.611989    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:40.612616 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:40.612010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:40.612616 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:40.612064    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:40.612616 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:40.612124    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:40.774409 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:40.774373    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj" event={"ID":"f478498d-4adf-441d-9dc6-1988d7c9e050","Type":"ContainerStarted","Data":"6dc4257fa18957c3538d6dede9a9305d5005369ef10780bce9a6897f6306b9c7"}
Oct 30 18:38:40.774646 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:40.774631    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:40.790703 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:40.790672    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:40.936572 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:40.936473    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj" podStartSLOduration=6.271081646 podStartE2EDuration="30.936457477s" totalImagesPullingTime="24.665375831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:40.93592503 +0000 UTC m=+31.847200495" watchObservedRunningTime="2026-10-30 18:38:40.936457477 +0000 UTC m=+31.847732941"
Oct 30 18:38:41.055592 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:41.055554    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:41.776785 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:41.776755    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:41.792018 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:41.791990    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:38:42.514569 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:42.514533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:42.514569 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:42.514571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:42.514780 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:42.514673    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:42.514780 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:42.514682    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:42.514780 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:42.514731    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret podName:ab2168bb-3bf1-48ed-993d-4d0a014f00ae nodeName:}" failed. No retries permitted until 2026-10-30 18:39:14.514716942 +0000 UTC m=+65.425992384 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret") pod "global-pull-secret-syncer-7h2cm" (UID: "ab2168bb-3bf1-48ed-993d-4d0a014f00ae") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:42.514780 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:42.514743    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs podName:e8659e3f-b750-4d38-aa0b-7cc7f883f9b4 nodeName:}" failed. No retries permitted until 2026-10-30 18:39:14.514736944 +0000 UTC m=+65.426012388 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs") pod "network-metrics-daemon-676kr" (UID: "e8659e3f-b750-4d38-aa0b-7cc7f883f9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:42.612087 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:42.612055    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:42.612283 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:42.612055    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:42.612283 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:42.612191    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:42.612283 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:42.612055    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:42.612283 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:42.612273    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:42.612421 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:42.612315    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:42.615234 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:42.615207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8lg\" (UniqueName: \"kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg\") pod \"network-check-target-27m75\" (UID: \"761ddebb-9140-4814-8145-3bb12ce8a148\") " pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:42.615375 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:42.615359    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:42.615413 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:42.615381    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:42.615413 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:42.615390    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4r8lg for pod openshift-network-diagnostics/network-check-target-27m75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:42.615498 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:42.615430    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg podName:761ddebb-9140-4814-8145-3bb12ce8a148 nodeName:}" failed. No retries permitted until 2026-10-30 18:39:14.615417416 +0000 UTC m=+65.526692863 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4r8lg" (UniqueName: "kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg") pod "network-check-target-27m75" (UID: "761ddebb-9140-4814-8145-3bb12ce8a148") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:43.099567 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:43.099376    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7h2cm"]
Oct 30 18:38:43.099977 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:43.099676    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:43.099977 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:43.099763    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:43.105851 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:43.105822    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-27m75"]
Oct 30 18:38:43.106113 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:43.106093    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:43.106232 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:43.106214    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:43.106696 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:43.106673    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-676kr"]
Oct 30 18:38:43.106802 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:43.106788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:43.106920 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:43.106898    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:44.611486 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:44.611453    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:44.611868 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:44.611555    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:44.611868 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:44.611559    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:44.611868 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:44.611587    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:44.611868 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:44.611673    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:44.611868 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:44.611721    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:46.611718 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:46.611680    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:46.612091 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:46.611681    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:46.612091 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:46.611786    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27m75" podUID="761ddebb-9140-4814-8145-3bb12ce8a148"
Oct 30 18:38:46.612091 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:46.611681    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:46.612091 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:46.611872    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-676kr" podUID="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4"
Oct 30 18:38:46.612091 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:38:46.611938    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7h2cm" podUID="ab2168bb-3bf1-48ed-993d-4d0a014f00ae"
Oct 30 18:38:48.342078 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.342050    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeReady"
Oct 30 18:38:48.342447 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.342218    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:38:48.497498 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.497454    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7rxxp"]
Oct 30 18:38:48.498944 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.498928    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fzvff"]
Oct 30 18:38:48.499082 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.499064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7rxxp"
Oct 30 18:38:48.500564 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.500546    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.502123 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.502107    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4kspp"]
Oct 30 18:38:48.503516 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.503497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:48.514822 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.514799    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7rxxp"]
Oct 30 18:38:48.552069 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.551985    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:38:48.552069 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.552034    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:38:48.552069 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.552066    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dx4x5\""
Oct 30 18:38:48.552315 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.552125    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:38:48.552315 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.552288    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:38:48.552445 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.552430    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:38:48.552498 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.552442    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bhm78\""
Oct 30 18:38:48.552690 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.552678    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:38:48.552721 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.552683    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:38:48.554338 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.554323    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:38:48.563699 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.563681    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:38:48.563699 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.563687    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xhgds\""
Oct 30 18:38:48.601831 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.601794    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4kspp"]
Oct 30 18:38:48.602343 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.602324    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fzvff"]
Oct 30 18:38:48.611756 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.611731    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:38:48.611872 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.611832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:38:48.611872 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.611733    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:38:48.661994 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.661956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.661994 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.661996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.662250 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.662021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf7720c-65a8-494e-a679-239f3a59ba3f-tmp-dir\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:48.662250 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.662092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf7720c-65a8-494e-a679-239f3a59ba3f-config-volume\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:48.662250 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.662129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdf7720c-65a8-494e-a679-239f3a59ba3f-metrics-tls\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:48.662250 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.662181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvfqq\" (UniqueName: \"kubernetes.io/projected/0b629f13-9f3f-41a9-9122-5d9fbe050beb-kube-api-access-vvfqq\") pod \"ingress-canary-7rxxp\" (UID: \"0b629f13-9f3f-41a9-9122-5d9fbe050beb\") " pod="openshift-ingress-canary/ingress-canary-7rxxp"
Oct 30 18:38:48.662250 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.662234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hqnx\" (UniqueName: \"kubernetes.io/projected/bdf7720c-65a8-494e-a679-239f3a59ba3f-kube-api-access-5hqnx\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:48.662409 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.662260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-data-volume\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.662409 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.662321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhvz6\" (UniqueName: \"kubernetes.io/projected/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-kube-api-access-qhvz6\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.662409 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.662346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-crio-socket\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.662409 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.662366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b629f13-9f3f-41a9-9122-5d9fbe050beb-cert\") pod \"ingress-canary-7rxxp\" (UID: \"0b629f13-9f3f-41a9-9122-5d9fbe050beb\") " pod="openshift-ingress-canary/ingress-canary-7rxxp"
Oct 30 18:38:48.663784 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.663766    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:38:48.663972 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.663958    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6nrlk\""
Oct 30 18:38:48.664057 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.663965    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:38:48.664057 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.663994    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-cts7j\""
Oct 30 18:38:48.664057 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.664003    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:38:48.664057 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.664021    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9gxzb\""
Oct 30 18:38:48.664216 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.664069    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:38:48.763002 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.762966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf7720c-65a8-494e-a679-239f3a59ba3f-config-volume\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:48.763002 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdf7720c-65a8-494e-a679-239f3a59ba3f-metrics-tls\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:48.763279 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvfqq\" (UniqueName: \"kubernetes.io/projected/0b629f13-9f3f-41a9-9122-5d9fbe050beb-kube-api-access-vvfqq\") pod \"ingress-canary-7rxxp\" (UID: \"0b629f13-9f3f-41a9-9122-5d9fbe050beb\") " pod="openshift-ingress-canary/ingress-canary-7rxxp"
Oct 30 18:38:48.763279 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqnx\" (UniqueName: \"kubernetes.io/projected/bdf7720c-65a8-494e-a679-239f3a59ba3f-kube-api-access-5hqnx\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:48.763279 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-data-volume\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.763279 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhvz6\" (UniqueName: \"kubernetes.io/projected/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-kube-api-access-qhvz6\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.763279 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-crio-socket\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.763279 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b629f13-9f3f-41a9-9122-5d9fbe050beb-cert\") pod \"ingress-canary-7rxxp\" (UID: \"0b629f13-9f3f-41a9-9122-5d9fbe050beb\") " pod="openshift-ingress-canary/ingress-canary-7rxxp"
Oct 30 18:38:48.763574 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.763574 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.763574 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf7720c-65a8-494e-a679-239f3a59ba3f-tmp-dir\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:48.763574 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763443    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-data-volume\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.763574 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763517    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-crio-socket\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.763776 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763696    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf7720c-65a8-494e-a679-239f3a59ba3f-tmp-dir\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:48.763776 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763732    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf7720c-65a8-494e-a679-239f3a59ba3f-config-volume\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:48.763776 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.763763    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.767038 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.767011    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdf7720c-65a8-494e-a679-239f3a59ba3f-metrics-tls\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:48.767164 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.767044    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.767164 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.767105    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b629f13-9f3f-41a9-9122-5d9fbe050beb-cert\") pod \"ingress-canary-7rxxp\" (UID: \"0b629f13-9f3f-41a9-9122-5d9fbe050beb\") " pod="openshift-ingress-canary/ingress-canary-7rxxp"
Oct 30 18:38:48.819775 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.819746    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhvz6\" (UniqueName: \"kubernetes.io/projected/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-kube-api-access-qhvz6\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:48.819925 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.819804    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqnx\" (UniqueName: \"kubernetes.io/projected/bdf7720c-65a8-494e-a679-239f3a59ba3f-kube-api-access-5hqnx\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:48.819965 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.819942    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvfqq\" (UniqueName: \"kubernetes.io/projected/0b629f13-9f3f-41a9-9122-5d9fbe050beb-kube-api-access-vvfqq\") pod \"ingress-canary-7rxxp\" (UID: \"0b629f13-9f3f-41a9-9122-5d9fbe050beb\") " pod="openshift-ingress-canary/ingress-canary-7rxxp"
Oct 30 18:38:48.823708 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:48.823688    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:49.110916 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:49.110699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7rxxp"
Oct 30 18:38:49.120310 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:49.120286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:38:49.200653 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:49.200623    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4kspp"]
Oct 30 18:38:49.203859 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:49.203836    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf7720c_65a8_494e_a679_239f3a59ba3f.slice/crio-6eee339ba2566a84033aef619ba3119c5f5388d7af8d24b1e9fe2eaa14e9bcde WatchSource:0}: Error finding container 6eee339ba2566a84033aef619ba3119c5f5388d7af8d24b1e9fe2eaa14e9bcde: Status 404 returned error can't find the container with id 6eee339ba2566a84033aef619ba3119c5f5388d7af8d24b1e9fe2eaa14e9bcde
Oct 30 18:38:49.480868 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:49.480794    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7rxxp"]
Oct 30 18:38:49.483659 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:49.483632    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b629f13_9f3f_41a9_9122_5d9fbe050beb.slice/crio-53c7757eaac07f8ff12d7f43254531484e914baefabf34d7db6d7eb0e5b28e36 WatchSource:0}: Error finding container 53c7757eaac07f8ff12d7f43254531484e914baefabf34d7db6d7eb0e5b28e36: Status 404 returned error can't find the container with id 53c7757eaac07f8ff12d7f43254531484e914baefabf34d7db6d7eb0e5b28e36
Oct 30 18:38:49.492646 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:49.492604    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fzvff"]
Oct 30 18:38:49.496173 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:49.496131    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac09d41c_10f4_4d5d_a8cb_1e638a70ab35.slice/crio-02aac1802c76057fad5ba627d7cd45c9b14cb558382cbced4aca6ab3663abd8e WatchSource:0}: Error finding container 02aac1802c76057fad5ba627d7cd45c9b14cb558382cbced4aca6ab3663abd8e: Status 404 returned error can't find the container with id 02aac1802c76057fad5ba627d7cd45c9b14cb558382cbced4aca6ab3663abd8e
Oct 30 18:38:49.798498 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:49.798405    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fzvff" event={"ID":"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35","Type":"ContainerStarted","Data":"4078f498509ed936d82903f01abcfeba70fbdbfceb7f09687c510a8401f8b6a0"}
Oct 30 18:38:49.798498 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:49.798468    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fzvff" event={"ID":"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35","Type":"ContainerStarted","Data":"02aac1802c76057fad5ba627d7cd45c9b14cb558382cbced4aca6ab3663abd8e"}
Oct 30 18:38:49.799499 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:49.799470    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7rxxp" event={"ID":"0b629f13-9f3f-41a9-9122-5d9fbe050beb","Type":"ContainerStarted","Data":"53c7757eaac07f8ff12d7f43254531484e914baefabf34d7db6d7eb0e5b28e36"}
Oct 30 18:38:49.800445 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:49.800422    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4kspp" event={"ID":"bdf7720c-65a8-494e-a679-239f3a59ba3f","Type":"ContainerStarted","Data":"6eee339ba2566a84033aef619ba3119c5f5388d7af8d24b1e9fe2eaa14e9bcde"}
Oct 30 18:38:50.807247 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:50.806020    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fzvff" event={"ID":"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35","Type":"ContainerStarted","Data":"92646519f37d7c36ef91219a8d63706446c5e7c6a3c5915b3d2886de02fe33c2"}
Oct 30 18:38:51.811669 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:51.811401    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4kspp" event={"ID":"bdf7720c-65a8-494e-a679-239f3a59ba3f","Type":"ContainerStarted","Data":"de51cf5ac571fa4a8e8f2fb09a03a8eff15304bb0c374843c21318ec42c6e76c"}
Oct 30 18:38:51.811669 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:51.811622    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4kspp" event={"ID":"bdf7720c-65a8-494e-a679-239f3a59ba3f","Type":"ContainerStarted","Data":"27df50b0fd6651e5d98a76880102900975ce7d9ae1cfdc7508bb14ac7acd7a12"}
Oct 30 18:38:51.812235 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:51.811747    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4kspp"
Oct 30 18:38:52.325018 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.324954    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-4kspp" podStartSLOduration=1.977508397 podStartE2EDuration="4.324929192s" totalImagesPullingTime="2.347420795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:51.928847955 +0000 UTC m=+42.840123419" watchObservedRunningTime="2026-10-30 18:38:52.324929192 +0000 UTC m=+43.236204658"
Oct 30 18:38:52.325358 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.325333    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"]
Oct 30 18:38:52.327564 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.327542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:52.380177 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.380081    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:38:52.380177 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.380124    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 18:38:52.428812 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.428778    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 18:38:52.428812 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.428788    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:38:52.429059 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.428845    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-w27xd\""
Oct 30 18:38:52.429059 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.428918    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:38:52.430453 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.430423    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"]
Oct 30 18:38:52.480318 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.480282    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h2x92"]
Oct 30 18:38:52.482509 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.482471    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"]
Oct 30 18:38:52.482808 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.482780    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.484680 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.484657    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.494029 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.493795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/717a2238-075b-4288-9c74-1a306efb331d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:52.494029 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.493832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhfj5\" (UniqueName: \"kubernetes.io/projected/717a2238-075b-4288-9c74-1a306efb331d-kube-api-access-xhfj5\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:52.494029 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.493856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/717a2238-075b-4288-9c74-1a306efb331d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:52.494029 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.493923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/717a2238-075b-4288-9c74-1a306efb331d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:52.523015 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.522984    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"]
Oct 30 18:38:52.549159 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.548929    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bh5pq\""
Oct 30 18:38:52.549159 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.548993    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xqkxc\""
Oct 30 18:38:52.549159 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.549029    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 18:38:52.549159 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.549042    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:38:52.549159 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.549059    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 18:38:52.549523 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.549207    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:38:52.594660 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.594618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/717a2238-075b-4288-9c74-1a306efb331d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:52.594830 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.594673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-wtmp\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.594830 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.594701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-sys\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.595507 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595478    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/717a2238-075b-4288-9c74-1a306efb331d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:52.595638 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.595638 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-textfile\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.595638 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-metrics-client-ca\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.595797 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-root\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.595797 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a1ed51f8-138a-4e90-bcfe-00cae5eba235-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.595797 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/717a2238-075b-4288-9c74-1a306efb331d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:52.595797 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-accelerators-collector-config\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.595797 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed51f8-138a-4e90-bcfe-00cae5eba235-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.595943 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6vzs\" (UniqueName: \"kubernetes.io/projected/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-kube-api-access-b6vzs\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.595943 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/717a2238-075b-4288-9c74-1a306efb331d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:52.595943 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhfj5\" (UniqueName: \"kubernetes.io/projected/717a2238-075b-4288-9c74-1a306efb331d-kube-api-access-xhfj5\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:52.595943 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.595943 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-tls\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.595943 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.595943 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.596235 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.595951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkl46\" (UniqueName: \"kubernetes.io/projected/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-api-access-gkl46\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.600557 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.600524    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/717a2238-075b-4288-9c74-1a306efb331d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:52.600557 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.600543    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/717a2238-075b-4288-9c74-1a306efb331d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:52.629458 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.629424    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:38:52.629636 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.629553    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 18:38:52.653102 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.653070    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhfj5\" (UniqueName: \"kubernetes.io/projected/717a2238-075b-4288-9c74-1a306efb331d-kube-api-access-xhfj5\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:52.696569 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkl46\" (UniqueName: \"kubernetes.io/projected/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-api-access-gkl46\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.696664 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-wtmp\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.696664 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-sys\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.696664 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.696808 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-textfile\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.696808 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-metrics-client-ca\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.696808 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-root\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.696808 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696731    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-sys\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.696808 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a1ed51f8-138a-4e90-bcfe-00cae5eba235-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.696808 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-accelerators-collector-config\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.697130 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696820    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-wtmp\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.697130 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed51f8-138a-4e90-bcfe-00cae5eba235-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.697130 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6vzs\" (UniqueName: \"kubernetes.io/projected/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-kube-api-access-b6vzs\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.697130 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.697130 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.696977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-tls\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.697130 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.697000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.697130 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.697027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.697457 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.697226    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a1ed51f8-138a-4e90-bcfe-00cae5eba235-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.697457 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.697285    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-root\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.697457 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.697369    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-metrics-client-ca\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.697598 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.697538    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-textfile\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.697598 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.697564    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed51f8-138a-4e90-bcfe-00cae5eba235-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.697669 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.697650    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-accelerators-collector-config\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.697862 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.697837    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.700133 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.700105    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.700290 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.700275    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.700895 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.700875    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-tls\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.700981 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.700941    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.752803 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.752775    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6vzs\" (UniqueName: \"kubernetes.io/projected/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-kube-api-access-b6vzs\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.752904 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.752842    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkl46\" (UniqueName: \"kubernetes.io/projected/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-api-access-gkl46\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.800585 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.800523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:38:52.807161 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.807120    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:38:52.809004 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:52.808975    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d438d4b_2388_4a65_a199_8cb4b3d27dc2.slice/crio-25139100cca5312773f20fc48d48fd08d0bd3c95aff49b9774511088baa3b43b WatchSource:0}: Error finding container 25139100cca5312773f20fc48d48fd08d0bd3c95aff49b9774511088baa3b43b: Status 404 returned error can't find the container with id 25139100cca5312773f20fc48d48fd08d0bd3c95aff49b9774511088baa3b43b
Oct 30 18:38:52.816072 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.815993    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fzvff" event={"ID":"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35","Type":"ContainerStarted","Data":"5b98d2458cf16cd68f7fbc40a1102da6c33debb44df9e9ce1f138ea2e399f62e"}
Oct 30 18:38:52.821473 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.821439    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7rxxp" event={"ID":"0b629f13-9f3f-41a9-9122-5d9fbe050beb","Type":"ContainerStarted","Data":"a6e90b3f76d13780ce1232d63217dae4140e56c1a338a36ec070387ebe49ba97"}
Oct 30 18:38:52.823011 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.822989    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h2x92" event={"ID":"5d438d4b-2388-4a65-a199-8cb4b3d27dc2","Type":"ContainerStarted","Data":"25139100cca5312773f20fc48d48fd08d0bd3c95aff49b9774511088baa3b43b"}
Oct 30 18:38:52.939708 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.939445    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fzvff" podStartSLOduration=1.9136637269999999 podStartE2EDuration="4.939423871s" totalImagesPullingTime="3.025760144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:52.938873475 +0000 UTC m=+43.850148940" watchObservedRunningTime="2026-10-30 18:38:52.939423871 +0000 UTC m=+43.850699338"
Oct 30 18:38:52.943000 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:52.942978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:38:53.051353 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:53.051209    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7rxxp" podStartSLOduration=1.809407837 podStartE2EDuration="5.051189662s" totalImagesPullingTime="3.241781825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:53.050251773 +0000 UTC m=+43.961527238" watchObservedRunningTime="2026-10-30 18:38:53.051189662 +0000 UTC m=+43.962465129"
Oct 30 18:38:53.234722 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:53.234691    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"]
Oct 30 18:38:53.237557 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:53.237523    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ed51f8_138a_4e90_bcfe_00cae5eba235.slice/crio-b5e91197db0c9ccd0a4eb969f4efe951a65f394cf376dd0af7e0bc9cf369a0a5 WatchSource:0}: Error finding container b5e91197db0c9ccd0a4eb969f4efe951a65f394cf376dd0af7e0bc9cf369a0a5: Status 404 returned error can't find the container with id b5e91197db0c9ccd0a4eb969f4efe951a65f394cf376dd0af7e0bc9cf369a0a5
Oct 30 18:38:53.335716 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:53.335671    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"]
Oct 30 18:38:53.339513 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:53.339488    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717a2238_075b_4288_9c74_1a306efb331d.slice/crio-597b9c55fb105ae5a89a46e99b6762bf9460c99371f2797b98afcc1442921283 WatchSource:0}: Error finding container 597b9c55fb105ae5a89a46e99b6762bf9460c99371f2797b98afcc1442921283: Status 404 returned error can't find the container with id 597b9c55fb105ae5a89a46e99b6762bf9460c99371f2797b98afcc1442921283
Oct 30 18:38:53.828631 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:53.828591    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt" event={"ID":"717a2238-075b-4288-9c74-1a306efb331d","Type":"ContainerStarted","Data":"8af19c476a2a5568c1df6b4641198482048675540a1022877f1f7518bbce29ac"}
Oct 30 18:38:53.829433 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:53.828657    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt" event={"ID":"717a2238-075b-4288-9c74-1a306efb331d","Type":"ContainerStarted","Data":"0e80eeab134fede12683fb4cf3d1f85d548d4b5f2d072593a8eb133474e62dc5"}
Oct 30 18:38:53.829433 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:53.828684    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt" event={"ID":"717a2238-075b-4288-9c74-1a306efb331d","Type":"ContainerStarted","Data":"597b9c55fb105ae5a89a46e99b6762bf9460c99371f2797b98afcc1442921283"}
Oct 30 18:38:53.830190 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:53.830119    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9" event={"ID":"a1ed51f8-138a-4e90-bcfe-00cae5eba235","Type":"ContainerStarted","Data":"b5e91197db0c9ccd0a4eb969f4efe951a65f394cf376dd0af7e0bc9cf369a0a5"}
Oct 30 18:38:54.609200 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.609165    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bbbd6864b-w27vc"]
Oct 30 18:38:54.613850 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.613821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.667744 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.667711    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 18:38:54.667744 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.667716    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 18:38:54.667918 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.667826    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wph9t\""
Oct 30 18:38:54.671666 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.671611    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bbbd6864b-w27vc"]
Oct 30 18:38:54.691848 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.691795    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:38:54.694768 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.694746    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.712622 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.712592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.712622 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.712634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.712836 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.712652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.712836 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.712677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.712836 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.712693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lpn9\" (UniqueName: \"kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.712836 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.712722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.712836 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.712782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73dec725-685b-4a6d-81b4-783f6c79494e-tmp\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.712836 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.712798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.760099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.760058    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 18:38:54.760099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.760079    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 18:38:54.760099 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.760100    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 18:38:54.760382 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.760187    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 18:38:54.760382 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.760221    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 18:38:54.811660 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.811622    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:38:54.813943 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.813914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.814079 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.813962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.814079 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.813990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.814079 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.814079 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-out\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.814079 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.814367 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.814367 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-web-config\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.814367 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpn9\" (UniqueName: \"kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.814367 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.814367 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.814367 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.814367 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.814367 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73dec725-685b-4a6d-81b4-783f6c79494e-tmp\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.814367 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.814765 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.814765 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.814765 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.814765 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.814765 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf5zd\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-kube-api-access-bf5zd\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.814765 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.814520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.815414 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.815387    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.817330 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.817171    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.818101 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.817774    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73dec725-685b-4a6d-81b4-783f6c79494e-tmp\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.818101 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.818062    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.820585 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.820560    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.820684 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.820631    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.826558 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.826535    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 18:38:54.826653 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.826535    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 18:38:54.826653 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.826582    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 18:38:54.826653 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.826640    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 18:38:54.826737 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.826661    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 18:38:54.826737 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.826703    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 18:38:54.826737 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.826704    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 18:38:54.826737 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.826720    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 18:38:54.826884 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.826846    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bftj9\""
Oct 30 18:38:54.826993 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.826957    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 18:38:54.826993 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.826983    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 18:38:54.828765 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.828744    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.835048 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.835019    2578 generic.go:356] "Generic (PLEG): container finished" podID="5d438d4b-2388-4a65-a199-8cb4b3d27dc2" containerID="3ef2e2ef41957189b82a5df17679842dc281332bd5b5eb6948b9abd106615c9b" exitCode=0
Oct 30 18:38:54.835203 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.835103    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h2x92" event={"ID":"5d438d4b-2388-4a65-a199-8cb4b3d27dc2","Type":"ContainerDied","Data":"3ef2e2ef41957189b82a5df17679842dc281332bd5b5eb6948b9abd106615c9b"}
Oct 30 18:38:54.875086 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.875022    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lpn9\" (UniqueName: \"kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.915604 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.915567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.915760 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.915617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-web-config\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.915760 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.915649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.915760 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.915695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.915893 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.915789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.915893 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.915823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.915893 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.915873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.916107 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.915903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.916107 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.915926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf5zd\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-kube-api-access-bf5zd\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.916107 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.915953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.916107 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.915976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.916107 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.916018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.916977 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.916588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-out\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.917722 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.917684    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.918117 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.918093    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.918229 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.918204    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.920653 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.920620    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.921451 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.921181    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.921451 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.921413    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-out\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.921572 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.921537    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.921872 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.921842    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.922248 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.922219    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.922328 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.922289    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.923927 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.923909    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.924753 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.924734    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-web-config\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:54.929105 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.929074    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:38:54.976762 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:54.975608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf5zd\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-kube-api-access-bf5zd\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:55.014936 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.014910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:55.018797 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.018771    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"]
Oct 30 18:38:55.020923 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.020896    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.077321 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.075981    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 18:38:55.077321 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.076078    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 18:38:55.077321 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.076187    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 18:38:55.077321 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.076260    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 18:38:55.077321 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.076339    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8668tapl4mk7f\""
Oct 30 18:38:55.077321 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.076380    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 18:38:55.077321 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.076488    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xnv76\""
Oct 30 18:38:55.092444 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.092415    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"]
Oct 30 18:38:55.118823 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.118799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6cfb91c-11df-4d59-b723-320754d81b99-tmp\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.118889 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.118835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xs2k\" (UniqueName: \"kubernetes.io/projected/a6cfb91c-11df-4d59-b723-320754d81b99-kube-api-access-5xs2k\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.118889 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.118853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-tls\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.118889 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.118880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6cfb91c-11df-4d59-b723-320754d81b99-metrics-client-ca\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.118980 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.118898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.119042 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.119007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.119219 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.119183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.119274 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.119260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.119312 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.119288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-grpc-tls\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.219839 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.219813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.220001 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.219855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.220001 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.219873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-grpc-tls\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.220001 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.219896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6cfb91c-11df-4d59-b723-320754d81b99-tmp\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.220001 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.219923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xs2k\" (UniqueName: \"kubernetes.io/projected/a6cfb91c-11df-4d59-b723-320754d81b99-kube-api-access-5xs2k\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.220001 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.219942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-tls\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.220001 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.219972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6cfb91c-11df-4d59-b723-320754d81b99-metrics-client-ca\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.220001 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.220000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.220387 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.220056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.220672 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.220606    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6cfb91c-11df-4d59-b723-320754d81b99-tmp\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.221111 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.221068    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6cfb91c-11df-4d59-b723-320754d81b99-metrics-client-ca\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.223073 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.223051    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.223188 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.223070    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-grpc-tls\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.223245 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.223190    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.223245 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.223199    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-tls\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.223811 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.223781    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.223903 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.223810    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.278390 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.278361    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xs2k\" (UniqueName: \"kubernetes.io/projected/a6cfb91c-11df-4d59-b723-320754d81b99-kube-api-access-5xs2k\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.310652 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.310621    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bbbd6864b-w27vc"]
Oct 30 18:38:55.313956 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:55.313922    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73dec725_685b_4a6d_81b4_783f6c79494e.slice/crio-80f03a6c2a94341aabe2204002e5b372c379b9f548b9d7e725600f186df1106c WatchSource:0}: Error finding container 80f03a6c2a94341aabe2204002e5b372c379b9f548b9d7e725600f186df1106c: Status 404 returned error can't find the container with id 80f03a6c2a94341aabe2204002e5b372c379b9f548b9d7e725600f186df1106c
Oct 30 18:38:55.351680 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.351648    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:38:55.466753 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.466719    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:38:55.469752 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:55.469720    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f54b22_ebcb_43e3_8a45_d61242595fca.slice/crio-cf8f58ce264a062b7151255601a6f5aab15ef00cc42e399e302fa0d279e113f9 WatchSource:0}: Error finding container cf8f58ce264a062b7151255601a6f5aab15ef00cc42e399e302fa0d279e113f9: Status 404 returned error can't find the container with id cf8f58ce264a062b7151255601a6f5aab15ef00cc42e399e302fa0d279e113f9
Oct 30 18:38:55.723008 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.722930    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"]
Oct 30 18:38:55.725962 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:55.725931    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6cfb91c_11df_4d59_b723_320754d81b99.slice/crio-19e8bbf907ef6a3653fb434d65382689fae6823d7043a464362050ee0028f4c1 WatchSource:0}: Error finding container 19e8bbf907ef6a3653fb434d65382689fae6823d7043a464362050ee0028f4c1: Status 404 returned error can't find the container with id 19e8bbf907ef6a3653fb434d65382689fae6823d7043a464362050ee0028f4c1
Oct 30 18:38:55.839949 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.839898    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7" event={"ID":"a6cfb91c-11df-4d59-b723-320754d81b99","Type":"ContainerStarted","Data":"19e8bbf907ef6a3653fb434d65382689fae6823d7043a464362050ee0028f4c1"}
Oct 30 18:38:55.841247 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.841224    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerStarted","Data":"cf8f58ce264a062b7151255601a6f5aab15ef00cc42e399e302fa0d279e113f9"}
Oct 30 18:38:55.843469 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.843439    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h2x92" event={"ID":"5d438d4b-2388-4a65-a199-8cb4b3d27dc2","Type":"ContainerStarted","Data":"e72ed23e7e72c91dea5cdbe35d83ec9d205dba7dd979eca376e602b197da4dc3"}
Oct 30 18:38:55.843561 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.843491    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h2x92" event={"ID":"5d438d4b-2388-4a65-a199-8cb4b3d27dc2","Type":"ContainerStarted","Data":"02ada739ba56bfa65ec32195676beb73b34d62826ff02e9c191ca202f752400c"}
Oct 30 18:38:55.845667 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.845639    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt" event={"ID":"717a2238-075b-4288-9c74-1a306efb331d","Type":"ContainerStarted","Data":"a02465310098c4e88e75cb7a595527cd4355d3360c891d1f9260c5b59c16a0df"}
Oct 30 18:38:55.848048 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.848026    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9" event={"ID":"a1ed51f8-138a-4e90-bcfe-00cae5eba235","Type":"ContainerStarted","Data":"eed8e458d9f256c6315c29d444df224e05dc8ad3b58cb6a716f05c814eea3a94"}
Oct 30 18:38:55.848226 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.848207    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9" event={"ID":"a1ed51f8-138a-4e90-bcfe-00cae5eba235","Type":"ContainerStarted","Data":"c691ea0873785dd6d4096a118f87af3c0dcb7a827f8e0b3b5dafe8192087290b"}
Oct 30 18:38:55.848308 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.848293    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9" event={"ID":"a1ed51f8-138a-4e90-bcfe-00cae5eba235","Type":"ContainerStarted","Data":"d95aa65d2416f5cfb1cb2980179193d135afd17b98a91c3b3d554de9a2c292fd"}
Oct 30 18:38:55.849676 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.849649    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbbd6864b-w27vc" event={"ID":"73dec725-685b-4a6d-81b4-783f6c79494e","Type":"ContainerStarted","Data":"80f03a6c2a94341aabe2204002e5b372c379b9f548b9d7e725600f186df1106c"}
Oct 30 18:38:55.958252 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:55.958197    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h2x92" podStartSLOduration=2.596562497 podStartE2EDuration="3.958181581s" totalImagesPullingTime="1.28677848s" totalInitContainerRuntime="74.840604ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:55.956882443 +0000 UTC m=+46.868157909" watchObservedRunningTime="2026-10-30 18:38:55.958181581 +0000 UTC m=+46.869457046"
Oct 30 18:38:56.073514 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:56.073459    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt" podStartSLOduration=2.527729922 podStartE2EDuration="4.073441589s" totalImagesPullingTime="1.545711667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:56.072114893 +0000 UTC m=+46.983390357" watchObservedRunningTime="2026-10-30 18:38:56.073441589 +0000 UTC m=+46.984717054"
Oct 30 18:38:56.186626 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:56.186530    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9" podStartSLOduration=2.374583429 podStartE2EDuration="4.186510397s" totalImagesPullingTime="1.811926968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:56.184993488 +0000 UTC m=+47.096268960" watchObservedRunningTime="2026-10-30 18:38:56.186510397 +0000 UTC m=+47.097785860"
Oct 30 18:38:57.140286 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.140250    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"]
Oct 30 18:38:57.142720 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.142693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:38:57.194572 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.194532    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 18:38:57.194572 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.194532    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zdt4f\""
Oct 30 18:38:57.200883 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.200856    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"]
Oct 30 18:38:57.237714 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.237634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg7ts\" (UniqueName: \"kubernetes.io/projected/909bd6eb-4140-4939-a5c1-4d148b3791dc-kube-api-access-qg7ts\") pod \"monitoring-plugin-77f4f76594-gbbdc\" (UID: \"909bd6eb-4140-4939-a5c1-4d148b3791dc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:38:57.237714 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.237687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909bd6eb-4140-4939-a5c1-4d148b3791dc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gbbdc\" (UID: \"909bd6eb-4140-4939-a5c1-4d148b3791dc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:38:57.245186 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.243851    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7848bc9d85-wmsb2"]
Oct 30 18:38:57.246492 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.246468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.298575 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.298541    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 18:38:57.298743 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.298598    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 18:38:57.298743 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.298619    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 18:38:57.298743 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.298675    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 18:38:57.298743 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.298703    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2obehff8v19io\""
Oct 30 18:38:57.298954 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.298780    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8wzk6\""
Oct 30 18:38:57.309808 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.309741    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7848bc9d85-wmsb2"]
Oct 30 18:38:57.338485 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.338453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cebbc4c-d198-4222-968d-4ab6d035ee51-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.338678 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.338545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg7ts\" (UniqueName: \"kubernetes.io/projected/909bd6eb-4140-4939-a5c1-4d148b3791dc-kube-api-access-qg7ts\") pod \"monitoring-plugin-77f4f76594-gbbdc\" (UID: \"909bd6eb-4140-4939-a5c1-4d148b3791dc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:38:57.338678 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.338602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-secret-metrics-server-client-certs\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.338678 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.338670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-client-ca-bundle\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.338796 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.338715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6cebbc4c-d198-4222-968d-4ab6d035ee51-audit-log\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.338796 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.338755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909bd6eb-4140-4939-a5c1-4d148b3791dc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gbbdc\" (UID: \"909bd6eb-4140-4939-a5c1-4d148b3791dc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:38:57.338857 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.338797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-secret-metrics-server-tls\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.339000 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.338969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vd7v\" (UniqueName: \"kubernetes.io/projected/6cebbc4c-d198-4222-968d-4ab6d035ee51-kube-api-access-9vd7v\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.339086 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.339006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6cebbc4c-d198-4222-968d-4ab6d035ee51-metrics-server-audit-profiles\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.341780 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.341719    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909bd6eb-4140-4939-a5c1-4d148b3791dc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gbbdc\" (UID: \"909bd6eb-4140-4939-a5c1-4d148b3791dc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:38:57.395253 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.395218    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg7ts\" (UniqueName: \"kubernetes.io/projected/909bd6eb-4140-4939-a5c1-4d148b3791dc-kube-api-access-qg7ts\") pod \"monitoring-plugin-77f4f76594-gbbdc\" (UID: \"909bd6eb-4140-4939-a5c1-4d148b3791dc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:38:57.440275 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.440238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-secret-metrics-server-tls\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.440465 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.440341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vd7v\" (UniqueName: \"kubernetes.io/projected/6cebbc4c-d198-4222-968d-4ab6d035ee51-kube-api-access-9vd7v\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.440465 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.440373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6cebbc4c-d198-4222-968d-4ab6d035ee51-metrics-server-audit-profiles\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.440465 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.440429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cebbc4c-d198-4222-968d-4ab6d035ee51-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.440624 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.440465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-secret-metrics-server-client-certs\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.440624 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.440494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-client-ca-bundle\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.440624 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.440539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6cebbc4c-d198-4222-968d-4ab6d035ee51-audit-log\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.440973 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.440950    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6cebbc4c-d198-4222-968d-4ab6d035ee51-audit-log\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.441614 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.441593    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cebbc4c-d198-4222-968d-4ab6d035ee51-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.442217 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.441861    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6cebbc4c-d198-4222-968d-4ab6d035ee51-metrics-server-audit-profiles\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.445005 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.444939    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-secret-metrics-server-client-certs\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.445005 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.444979    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-secret-metrics-server-tls\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.445759 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.445734    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-client-ca-bundle\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.455649 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.455629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:38:57.499662 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.499587    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vd7v\" (UniqueName: \"kubernetes.io/projected/6cebbc4c-d198-4222-968d-4ab6d035ee51-kube-api-access-9vd7v\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.560402 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.560362    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:38:57.837063 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.837029    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"]
Oct 30 18:38:57.840221 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:57.840184    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909bd6eb_4140_4939_a5c1_4d148b3791dc.slice/crio-194f6900f57730baa032582caa26db939ed29174446349177f142f0bb636b8f9 WatchSource:0}: Error finding container 194f6900f57730baa032582caa26db939ed29174446349177f142f0bb636b8f9: Status 404 returned error can't find the container with id 194f6900f57730baa032582caa26db939ed29174446349177f142f0bb636b8f9
Oct 30 18:38:57.861061 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.861030    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerID="6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133" exitCode=0
Oct 30 18:38:57.861218 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.861177    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerDied","Data":"6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133"}
Oct 30 18:38:57.862841 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.862813    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc" event={"ID":"909bd6eb-4140-4939-a5c1-4d148b3791dc","Type":"ContainerStarted","Data":"194f6900f57730baa032582caa26db939ed29174446349177f142f0bb636b8f9"}
Oct 30 18:38:57.946881 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:57.946789    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7848bc9d85-wmsb2"]
Oct 30 18:38:57.950810 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:57.950784    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cebbc4c_d198_4222_968d_4ab6d035ee51.slice/crio-905a8c6384e5e7c3a1c0fd3d0cde3432a320af6d22d82b0d7a9b72bc7edc4614 WatchSource:0}: Error finding container 905a8c6384e5e7c3a1c0fd3d0cde3432a320af6d22d82b0d7a9b72bc7edc4614: Status 404 returned error can't find the container with id 905a8c6384e5e7c3a1c0fd3d0cde3432a320af6d22d82b0d7a9b72bc7edc4614
Oct 30 18:38:58.178700 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.177351    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-78cd67666d-td5fr"]
Oct 30 18:38:58.181284 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.180766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.233473 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.233418    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 18:38:58.233473 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.233418    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 18:38:58.233473 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.233418    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 18:38:58.233473 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.233418    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 18:38:58.233473 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.233466    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-m8bck\""
Oct 30 18:38:58.233835 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.233752    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 18:38:58.283091 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.283053    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 18:38:58.284645 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.284608    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78cd67666d-td5fr"]
Oct 30 18:38:58.349719 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.349667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-serving-certs-ca-bundle\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.349719 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.349725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjkqx\" (UniqueName: \"kubernetes.io/projected/2cf0ddaf-aa3c-4390-b911-cab27953af11-kube-api-access-cjkqx\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.349974 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.349776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-federate-client-tls\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.349974 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.349805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-secret-telemeter-client\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.349974 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.349825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.349974 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.349883    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-metrics-client-ca\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.349974 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.349904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.349974 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.349927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-telemeter-client-tls\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.451353 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.451277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-serving-certs-ca-bundle\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.451353 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.451317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjkqx\" (UniqueName: \"kubernetes.io/projected/2cf0ddaf-aa3c-4390-b911-cab27953af11-kube-api-access-cjkqx\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.451546 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.451452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-federate-client-tls\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.451546 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.451505    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-secret-telemeter-client\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.451617 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.451553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.451668 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.451643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-metrics-client-ca\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.451715 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.451684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.451765 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.451725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-telemeter-client-tls\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.452054 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.452027    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-serving-certs-ca-bundle\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.452513 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.452490    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-metrics-client-ca\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.453071 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.453045    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.455112 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.455091    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-secret-telemeter-client\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.455286 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.455254    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-telemeter-client-tls\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.455382 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.455299    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.455886 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.455863    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-federate-client-tls\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.514124 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.514085    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjkqx\" (UniqueName: \"kubernetes.io/projected/2cf0ddaf-aa3c-4390-b911-cab27953af11-kube-api-access-cjkqx\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.801186 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.801091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:38:58.872681 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:58.872648    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2" event={"ID":"6cebbc4c-d198-4222-968d-4ab6d035ee51","Type":"ContainerStarted","Data":"905a8c6384e5e7c3a1c0fd3d0cde3432a320af6d22d82b0d7a9b72bc7edc4614"}
Oct 30 18:38:59.308324 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:59.308246    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78cd67666d-td5fr"]
Oct 30 18:38:59.313645 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:38:59.313608    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf0ddaf_aa3c_4390_b911_cab27953af11.slice/crio-43ad8896fedb69b24c0b82cec0a408554d789a07c371cb6b4fefae7c658c4d6f WatchSource:0}: Error finding container 43ad8896fedb69b24c0b82cec0a408554d789a07c371cb6b4fefae7c658c4d6f: Status 404 returned error can't find the container with id 43ad8896fedb69b24c0b82cec0a408554d789a07c371cb6b4fefae7c658c4d6f
Oct 30 18:38:59.880282 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:59.880243    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7" event={"ID":"a6cfb91c-11df-4d59-b723-320754d81b99","Type":"ContainerStarted","Data":"112d0f00338accfeb99fb65765be523ea352368a8f2068f70b70fae37f69896e"}
Oct 30 18:38:59.880453 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:59.880332    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7" event={"ID":"a6cfb91c-11df-4d59-b723-320754d81b99","Type":"ContainerStarted","Data":"9d1898a58bb2d22c70a0cdad84f04ee2ac4e3bb7094bd8f66436fd503564c166"}
Oct 30 18:38:59.880453 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:59.880373    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7" event={"ID":"a6cfb91c-11df-4d59-b723-320754d81b99","Type":"ContainerStarted","Data":"fa9283d072b51fd577a5b6a29258f9c11749e574f2dcd68f6e0c873be8e74d94"}
Oct 30 18:38:59.881856 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:59.881819    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr" event={"ID":"2cf0ddaf-aa3c-4390-b911-cab27953af11","Type":"ContainerStarted","Data":"43ad8896fedb69b24c0b82cec0a408554d789a07c371cb6b4fefae7c658c4d6f"}
Oct 30 18:38:59.884291 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:59.884263    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbbd6864b-w27vc" event={"ID":"73dec725-685b-4a6d-81b4-783f6c79494e","Type":"ContainerStarted","Data":"84fccaea4bf5830aa95561b37f58f0989d173a527f1b7b83830fd475446883bd"}
Oct 30 18:38:59.999186 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:38:59.999056    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6bbbd6864b-w27vc" podStartSLOduration=2.496037072 podStartE2EDuration="5.999036492s" totalImagesPullingTime="3.50299942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:59.997327975 +0000 UTC m=+50.908603441" watchObservedRunningTime="2026-10-30 18:38:59.999036492 +0000 UTC m=+50.910311957"
Oct 30 18:39:00.891277 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.891179    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc" event={"ID":"909bd6eb-4140-4939-a5c1-4d148b3791dc","Type":"ContainerStarted","Data":"67d00adba5db3cd2ca451b37a2dd24a5c14091124a3b27790c757f605b31cf55"}
Oct 30 18:39:00.891722 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.891463    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:39:00.893747 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.893517    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2" event={"ID":"6cebbc4c-d198-4222-968d-4ab6d035ee51","Type":"ContainerStarted","Data":"0f2de03472073f72a30fa28b52d9b56d1ace9d3033de7776c57d1a1fcdb45a05"}
Oct 30 18:39:00.897283 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.897258    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7" event={"ID":"a6cfb91c-11df-4d59-b723-320754d81b99","Type":"ContainerStarted","Data":"db541d2424497dde623afb7ffd75b6a1e94c18c40389be8bf86adcb949e10847"}
Oct 30 18:39:00.897411 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.897326    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7" event={"ID":"a6cfb91c-11df-4d59-b723-320754d81b99","Type":"ContainerStarted","Data":"31b1a67f451ccec20133911ec89f4e14d4ebd1602e3ca8d8af16819df6a42112"}
Oct 30 18:39:00.897411 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.897374    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7" event={"ID":"a6cfb91c-11df-4d59-b723-320754d81b99","Type":"ContainerStarted","Data":"bf86d7b2e87a64af94c0db2f0c9afda6ba9e7ca65f09ae471dace75560cb076b"}
Oct 30 18:39:00.897482 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.897420    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:00.897653 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.897607    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:39:00.901542 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.901517    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerStarted","Data":"6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e"}
Oct 30 18:39:00.901653 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.901581    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerStarted","Data":"90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f"}
Oct 30 18:39:00.901653 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.901607    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerStarted","Data":"49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95"}
Oct 30 18:39:00.901653 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.901642    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerStarted","Data":"873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8"}
Oct 30 18:39:00.901820 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.901670    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerStarted","Data":"e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888"}
Oct 30 18:39:00.901820 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:00.901693    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerStarted","Data":"273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960"}
Oct 30 18:39:01.006613 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.006552    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc" podStartSLOduration=1.893451628 podStartE2EDuration="4.006536818s" totalImagesPullingTime="2.11308519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:01.005336576 +0000 UTC m=+51.916612042" watchObservedRunningTime="2026-10-30 18:39:01.006536818 +0000 UTC m=+51.917812282"
Oct 30 18:39:01.127418 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.127359    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.320708607 podStartE2EDuration="7.127340277s" totalImagesPullingTime="3.788060955s" totalInitContainerRuntime="18.570715ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:01.126644467 +0000 UTC m=+52.037919962" watchObservedRunningTime="2026-10-30 18:39:01.127340277 +0000 UTC m=+52.038615723"
Oct 30 18:39:01.246013 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.245908    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7" podStartSLOduration=2.738019201 podStartE2EDuration="7.245893774s" totalImagesPullingTime="4.507874573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:01.244659125 +0000 UTC m=+52.155934614" watchObservedRunningTime="2026-10-30 18:39:01.245893774 +0000 UTC m=+52.157169238"
Oct 30 18:39:01.350042 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.350007    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:01.354317 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.354289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.411106 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.410996    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 18:39:01.411283 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.411274    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 18:39:01.411369 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.411355    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 18:39:01.411421 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.411405    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 18:39:01.411887 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.411608    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lx4z8\""
Oct 30 18:39:01.411887 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.411606    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 18:39:01.411887 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.411633    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 18:39:01.411887 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.411614    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 18:39:01.411887 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.411629    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 18:39:01.411887 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.411635    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 18:39:01.411887 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.411614    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bdjh3qnjck7pd\""
Oct 30 18:39:01.411887 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.411786    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 18:39:01.431466 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.431438    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 18:39:01.431616 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.431438    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 18:39:01.461458 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.461419    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 18:39:01.461589 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.461552    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 18:39:01.473842 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.473817    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:01.481330 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.481486 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.481486 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.481486 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.481611 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.481611 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.481611 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.481750 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.481783 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtzzk\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-kube-api-access-qtzzk\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.481815 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-config\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.481865 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.481921 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.481958 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.482003 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.481964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.482044 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.482022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.482093 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.482067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.482132 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.482095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.482132 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.482120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.482217 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.482193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.482305 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.482284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.483900 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.483849    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2" podStartSLOduration=2.306861279 podStartE2EDuration="4.48383373s" totalImagesPullingTime="2.176972451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:01.481064682 +0000 UTC m=+52.392340149" watchObservedRunningTime="2026-10-30 18:39:01.48383373 +0000 UTC m=+52.395109193"
Oct 30 18:39:01.582913 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.582871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.582913 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.582905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583133 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.582921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583133 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.582936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583133 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.582961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583133 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583423 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583423 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583423 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583423 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583423 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583423 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583753 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583753 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583753 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583753 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583753 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583753 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtzzk\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-kube-api-access-qtzzk\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583753 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-config\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.583753 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.583661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.584574 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.584543    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.584828 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.584801    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.584981 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.584952    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.585276 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.585249    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.588564 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.587119    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.588564 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.587668    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.588564 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.587945    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.588564 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.588048    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.588564 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.588347    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.588854 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.588634    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.588854 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.588740    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.588854 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.588744    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.588854 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.588778    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.589049 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.588972    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.589207 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.589183    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.589423 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.589399    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.589508 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.589484    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.589591 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.589573    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.589823 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.589803    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-config\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.644942 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.644919    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtzzk\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-kube-api-access-qtzzk\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.682686 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.682473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:01.825688 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.825656    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4kspp"
Oct 30 18:39:01.906942 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.906865    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr" event={"ID":"2cf0ddaf-aa3c-4390-b911-cab27953af11","Type":"ContainerStarted","Data":"00716eedaf828e2cb866550504d229412952363b3773dc7a8be6c01744771906"}
Oct 30 18:39:01.906942 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.906929    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr" event={"ID":"2cf0ddaf-aa3c-4390-b911-cab27953af11","Type":"ContainerStarted","Data":"e405bca6447bfd32a464a9a1495be49eaf81bb7f71f92ea32ccd26215e80d6db"}
Oct 30 18:39:01.907359 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:01.906950    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr" event={"ID":"2cf0ddaf-aa3c-4390-b911-cab27953af11","Type":"ContainerStarted","Data":"749c45e6617a5ca0a36976024c43e4e5b8fc777516678a0f39a4cfc4a6fe9daa"}
Oct 30 18:39:02.057146 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:02.057091    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr" podStartSLOduration=1.7768708530000001 podStartE2EDuration="4.057075846s" totalImagesPullingTime="2.280204993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:02.054498385 +0000 UTC m=+52.965773874" watchObservedRunningTime="2026-10-30 18:39:02.057075846 +0000 UTC m=+52.968351311"
Oct 30 18:39:02.111560 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:02.111539    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:02.113664 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:39:02.113634    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd615363f_bfb2_4a9f_8678_774d3effd7d6.slice/crio-fe32fd6bee74600dc227871e74853cd90c58569364d9b319f3341099fe742d65 WatchSource:0}: Error finding container fe32fd6bee74600dc227871e74853cd90c58569364d9b319f3341099fe742d65: Status 404 returned error can't find the container with id fe32fd6bee74600dc227871e74853cd90c58569364d9b319f3341099fe742d65
Oct 30 18:39:02.916452 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:02.916411    2578 generic.go:356] "Generic (PLEG): container finished" podID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerID="004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced" exitCode=0
Oct 30 18:39:02.916898 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:02.916494    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerDied","Data":"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"}
Oct 30 18:39:02.916898 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:02.916579    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerStarted","Data":"fe32fd6bee74600dc227871e74853cd90c58569364d9b319f3341099fe742d65"}
Oct 30 18:39:04.666376 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:04.666337    2578 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bbbd6864b-w27vc"]
Oct 30 18:39:04.930018 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:04.929911    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:06.913339 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:06.913316    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:06.935598 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:06.935567    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerStarted","Data":"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"}
Oct 30 18:39:07.943150 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:07.943114    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerStarted","Data":"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"}
Oct 30 18:39:07.943547 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:07.943193    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerStarted","Data":"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"}
Oct 30 18:39:07.943547 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:07.943266    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerStarted","Data":"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"}
Oct 30 18:39:07.943547 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:07.943290    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerStarted","Data":"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"}
Oct 30 18:39:07.943547 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:07.943313    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerStarted","Data":"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"}
Oct 30 18:39:08.066371 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:08.066314    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.168488388 podStartE2EDuration="7.066301605s" totalImagesPullingTime="3.884017777s" totalInitContainerRuntime="13.79544ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:08.06605858 +0000 UTC m=+58.977334066" watchObservedRunningTime="2026-10-30 18:39:08.066301605 +0000 UTC m=+58.977577100"
Oct 30 18:39:11.682882 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:11.682850    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:13.792263 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:13.792235    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:14.515872 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.515828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:39:14.515872 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.515872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:39:14.567320 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.567282    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:39:14.567320 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.567282    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:39:14.579356 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.579320    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:39:14.579496 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.579339    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:39:14.617285 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.617251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8lg\" (UniqueName: \"kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg\") pod \"network-check-target-27m75\" (UID: \"761ddebb-9140-4814-8145-3bb12ce8a148\") " pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:39:14.668353 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.668321    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:39:14.729012 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.728980    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:39:14.741372 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.741345    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r8lg\" (UniqueName: \"kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg\") pod \"network-check-target-27m75\" (UID: \"761ddebb-9140-4814-8145-3bb12ce8a148\") " pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:39:14.774731 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.774659    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6nrlk\""
Oct 30 18:39:14.782599 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.782579    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-cts7j\""
Oct 30 18:39:14.784692 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.784678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:39:14.792269 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:14.792250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:39:15.077945 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:15.077918    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9gxzb\""
Oct 30 18:39:15.088179 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:15.088126    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:39:15.159287 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:15.159257    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-676kr"]
Oct 30 18:39:15.198409 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:39:15.198368    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8659e3f_b750_4d38_aa0b_7cc7f883f9b4.slice/crio-77b2e7f0cab9f9b65467f7a928c2a720a6a7c5842410a587abeae62562f77e41 WatchSource:0}: Error finding container 77b2e7f0cab9f9b65467f7a928c2a720a6a7c5842410a587abeae62562f77e41: Status 404 returned error can't find the container with id 77b2e7f0cab9f9b65467f7a928c2a720a6a7c5842410a587abeae62562f77e41
Oct 30 18:39:15.208645 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:15.208618    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7h2cm"]
Oct 30 18:39:15.211786 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:39:15.211764    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab2168bb_3bf1_48ed_993d_4d0a014f00ae.slice/crio-c137d5e1976d9fa3d8d22f7da32ff85d3f077cee7d00165161516afd1e0b1598 WatchSource:0}: Error finding container c137d5e1976d9fa3d8d22f7da32ff85d3f077cee7d00165161516afd1e0b1598: Status 404 returned error can't find the container with id c137d5e1976d9fa3d8d22f7da32ff85d3f077cee7d00165161516afd1e0b1598
Oct 30 18:39:15.455340 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:15.455252    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-27m75"]
Oct 30 18:39:15.458221 ip-10-0-143-97 kubenswrapper[2578]: W1030 18:39:15.458192    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761ddebb_9140_4814_8145_3bb12ce8a148.slice/crio-2f5f6e6a669926fd233d4e892d2014812bcb0dea5a09c85127946930f19648be WatchSource:0}: Error finding container 2f5f6e6a669926fd233d4e892d2014812bcb0dea5a09c85127946930f19648be: Status 404 returned error can't find the container with id 2f5f6e6a669926fd233d4e892d2014812bcb0dea5a09c85127946930f19648be
Oct 30 18:39:15.974720 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:15.974681    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-27m75" event={"ID":"761ddebb-9140-4814-8145-3bb12ce8a148","Type":"ContainerStarted","Data":"2f5f6e6a669926fd233d4e892d2014812bcb0dea5a09c85127946930f19648be"}
Oct 30 18:39:15.975883 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:15.975860    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7h2cm" event={"ID":"ab2168bb-3bf1-48ed-993d-4d0a014f00ae","Type":"ContainerStarted","Data":"c137d5e1976d9fa3d8d22f7da32ff85d3f077cee7d00165161516afd1e0b1598"}
Oct 30 18:39:15.976857 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:15.976836    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-676kr" event={"ID":"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4","Type":"ContainerStarted","Data":"77b2e7f0cab9f9b65467f7a928c2a720a6a7c5842410a587abeae62562f77e41"}
Oct 30 18:39:17.561586 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:17.561548    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:17.561996 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:17.561624    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:17.989318 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:17.989274    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-676kr" event={"ID":"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4","Type":"ContainerStarted","Data":"13789ee1f4e249700b03ad757ac890d30c3f20a6f22cbcbf222b7f1401fb3dc4"}
Oct 30 18:39:17.989506 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:17.989383    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-676kr" event={"ID":"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4","Type":"ContainerStarted","Data":"a1e4f284aee646ca58bd6ca876df9c6a50b4c0dc22501db8afd49c40c71434e8"}
Oct 30 18:39:18.106840 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:18.106736    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-676kr" podStartSLOduration=67.282032131 podStartE2EDuration="1m9.106717871s" totalImagesPullingTime="1.82468574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:18.105579238 +0000 UTC m=+69.016854704" watchObservedRunningTime="2026-10-30 18:39:18.106717871 +0000 UTC m=+69.017993338"
Oct 30 18:39:19.156015 ip-10-0-143-97 kubenswrapper[2578]: E1030 18:39:19.155980    2578 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:39:20.001004 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:20.000966    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-27m75" event={"ID":"761ddebb-9140-4814-8145-3bb12ce8a148","Type":"ContainerStarted","Data":"3d5cb518913fac50c2a5a3f663513a54fde1ac825d4929a577059d2cfb4fea52"}
Oct 30 18:39:20.001201 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:20.001093    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:39:20.119078 ip-10-0-143-97 kubenswrapper[2578]: I1030 18:39:20.119017    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-27m75" podStartSLOduration=67.292429853 podStartE2EDuration="1m11.118998326s" totalImagesPullingTime="3.826568473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:20.117641442 +0000 UTC m=+71.028916908" watchObservedRunningTime="2026-10-30 18:39:20.118998326 +0000 UTC m=+71.030273792"
Oct 30 18:39:21.281897 ip-10-0-143-97 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 18:39:21.301515 ip-10-0-143-97 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 18:39:21.301697 ip-10-0-143-97 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 18:39:21.301886 ip-10-0-143-97 systemd[1]: kubelet.service: Consumed 5.202s CPU time.
Oct 30 18:39:21.324025 ip-10-0-143-97 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:39:21.389212 ip-10-0-143-97 kubenswrapper[6014]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:39:21.389212 ip-10-0-143-97 kubenswrapper[6014]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:39:21.389212 ip-10-0-143-97 kubenswrapper[6014]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:39:21.389212 ip-10-0-143-97 kubenswrapper[6014]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:39:21.391362 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391269    6014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:21.391362 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391360    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:21.391362 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391364    6014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:21.391362 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391367    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391370    6014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391373    6014 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391376    6014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391379    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391385    6014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391388    6014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391391    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391394    6014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391396    6014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391399    6014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391402    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391404    6014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391407    6014 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391409    6014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391412    6014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391414    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391417    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391419    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391422    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:21.391484 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391424    6014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391428    6014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391431    6014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391433    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391436    6014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391439    6014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391441    6014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391444    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391446    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391449    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391451    6014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391454    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391457    6014 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391459    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391462    6014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391464    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391467    6014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391470    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391473    6014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:21.391982 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391475    6014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391479    6014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391484    6014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391487    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391490    6014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391493    6014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391496    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391498    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391501    6014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391503    6014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391506    6014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391509    6014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391512    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391515    6014 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391517    6014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391520    6014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391522    6014 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391525    6014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391528    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:21.392480 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391530    6014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391533    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391535    6014 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391538    6014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391540    6014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391543    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391545    6014 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391548    6014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391552    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391554    6014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391557    6014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391560    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391563    6014 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391566    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391569    6014 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391571    6014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391573    6014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391576    6014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391579    6014 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:21.392938 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391581    6014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391584    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391587    6014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391589    6014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391592    6014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391594    6014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391596    6014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391599    6014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391602    6014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391604    6014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391607    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391609    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391612    6014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391615    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391617    6014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391619    6014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391623    6014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391629    6014 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391632    6014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:21.393455 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391635    6014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391637    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391640    6014 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391642    6014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391644    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391647    6014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.391650    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392431    6014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392438    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392442    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392445    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392448    6014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392450    6014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392453    6014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392456    6014 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392458    6014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392461    6014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392463    6014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392466    6014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392468    6014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:21.393952 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392471    6014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392473    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392476    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392479    6014 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392482    6014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392485    6014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392488    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392491    6014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392494    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392496    6014 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392499    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392502    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392504    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392507    6014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392509    6014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392512    6014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392514    6014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392517    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392519    6014 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392522    6014 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:21.394462 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392525    6014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392527    6014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392530    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392533    6014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392535    6014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392537    6014 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392540    6014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392542    6014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392545    6014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392547    6014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392550    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392552    6014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392555    6014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392557    6014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392560    6014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392563    6014 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392565    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392568    6014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392571    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392573    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:21.394979 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392575    6014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392578    6014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392581    6014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392583    6014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392586    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392588    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392591    6014 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392594    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392596    6014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392599    6014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392601    6014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392604    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392606    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392609    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392612    6014 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392616    6014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392620    6014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392623    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392625    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:21.395559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392628    6014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392630    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392633    6014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392635    6014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392638    6014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392640    6014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392643    6014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392646    6014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392648    6014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392651    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392654    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392657    6014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392660    6014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392664    6014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392667    6014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392670    6014 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392673    6014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392676    6014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392678    6014 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:21.396056 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392681    6014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392684    6014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392686    6014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392688    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392691    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392693    6014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392696    6014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392707    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392710    6014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392713    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392715    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392718    6014 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392720    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392723    6014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.392725    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393230    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393236    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393238    6014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393241    6014 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:21.396580 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393244    6014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393247    6014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393249    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393252    6014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393255    6014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393261    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393264    6014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393266    6014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393269    6014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393272    6014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393274    6014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393277    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393280    6014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393284    6014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393287    6014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393290    6014 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393294    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393296    6014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393299    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:21.397046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393301    6014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393305    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393308    6014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393310    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393313    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393315    6014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393317    6014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393320    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393322    6014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393325    6014 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393327    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393330    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393334    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393337    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393339    6014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393342    6014 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393344    6014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393347    6014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393351    6014 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393354    6014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:21.397574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393357    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393359    6014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393362    6014 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393364    6014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393367    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393369    6014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393372    6014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393375    6014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393379    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393382    6014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393385    6014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393388    6014 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393390    6014 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393393    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393401    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393404    6014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393407    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393410    6014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393412    6014 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:21.398099 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393415    6014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393417    6014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393419    6014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393422    6014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393425    6014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393429    6014 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393431    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393434    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393436    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393439    6014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393441    6014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393445    6014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393448    6014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393450    6014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393453    6014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393455    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393458    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393461    6014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393463    6014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393466    6014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:21.398606 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393468    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393471    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393473    6014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393476    6014 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393478    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393480    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393483    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393485    6014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393488    6014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393491    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393493    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393496    6014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393498    6014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393500    6014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393503    6014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393505    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393507    6014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393511    6014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393513    6014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393516    6014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:21.399237 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393518    6014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393520    6014 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393524    6014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.393527    6014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393621    6014 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393628    6014 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393634    6014 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393639    6014 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393644    6014 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393647    6014 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393651    6014 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393655    6014 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393658    6014 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393661    6014 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393665    6014 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393668    6014 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393671    6014 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393674    6014 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393677    6014 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393680    6014 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393682    6014 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393686    6014 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393690    6014 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393693    6014 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:39:21.399966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393696    6014 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393699    6014 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393702    6014 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393706    6014 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393709    6014 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393711    6014 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393715    6014 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393717    6014 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393721    6014 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393724    6014 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393727    6014 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393730    6014 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393734    6014 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393737    6014 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393739    6014 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393742    6014 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393745    6014 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393748    6014 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393753    6014 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393756    6014 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393758    6014 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393761    6014 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393764    6014 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393768    6014 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:39:21.400609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393771    6014 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393774    6014 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393777    6014 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393780    6014 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393783    6014 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393786    6014 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393789    6014 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393792    6014 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393795    6014 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393798    6014 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393806    6014 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393809    6014 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393812    6014 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393815    6014 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393818    6014 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393821    6014 flags.go:64] FLAG: --help="false"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393824    6014 flags.go:64] FLAG: --hostname-override="ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393827    6014 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393830    6014 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393833    6014 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393836    6014 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393839    6014 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393842    6014 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:39:21.401313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393844    6014 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393847    6014 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393850    6014 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393853    6014 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393856    6014 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393858    6014 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393861    6014 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393864    6014 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393867    6014 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393870    6014 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393873    6014 flags.go:64] FLAG: --lock-file=""
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393877    6014 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393880    6014 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393883    6014 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393889    6014 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393892    6014 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393896    6014 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393899    6014 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393901    6014 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393905    6014 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393908    6014 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393911    6014 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393915    6014 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393919    6014 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393923    6014 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:39:21.401912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393926    6014 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393929    6014 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393933    6014 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393935    6014 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393938    6014 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393941    6014 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393944    6014 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393951    6014 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393955    6014 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393958    6014 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393961    6014 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393964    6014 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393967    6014 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393970    6014 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393973    6014 flags.go:64] FLAG: --port="10250"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393976    6014 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393979    6014 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-093c7716fc2eeca41"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393982    6014 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393986    6014 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393989    6014 flags.go:64] FLAG: --register-node="true"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393992    6014 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393996    6014 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.393999    6014 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394002    6014 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394005    6014 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394009    6014 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:39:21.402545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394012    6014 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394015    6014 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394018    6014 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394020    6014 flags.go:64] FLAG: --runonce="false"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394025    6014 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394028    6014 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394031    6014 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394034    6014 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394037    6014 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394040    6014 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394043    6014 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394046    6014 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394049    6014 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394052    6014 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394055    6014 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394058    6014 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394061    6014 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394063    6014 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394066    6014 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394072    6014 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394075    6014 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394077    6014 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394081    6014 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394084    6014 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394089    6014 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:39:21.403255 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394092    6014 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394095    6014 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394098    6014 flags.go:64] FLAG: --v="2"
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394102    6014 flags.go:64] FLAG: --version="false"
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394106    6014 flags.go:64] FLAG: --vmodule=""
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394110    6014 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394114    6014 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394278    6014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394282    6014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394285    6014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394288    6014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394292    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394295    6014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394297    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394300    6014 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394302    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394305    6014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394308    6014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394310    6014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394312    6014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394315    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:21.403878 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394318    6014 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394320    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394323    6014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394325    6014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394328    6014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394330    6014 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394333    6014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394335    6014 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394338    6014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394340    6014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394344    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394347    6014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394349    6014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394352    6014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394354    6014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394357    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394359    6014 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394362    6014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394370    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394373    6014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:21.404439 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394376    6014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394378    6014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394382    6014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394385    6014 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394387    6014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394390    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394392    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394395    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394397    6014 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394400    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394403    6014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394405    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394407    6014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394410    6014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394412    6014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394415    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394417    6014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394419    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394422    6014 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:21.404960 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394425    6014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394427    6014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394430    6014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394433    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394435    6014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394438    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394441    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394443    6014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394446    6014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394448    6014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394451    6014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394453    6014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394461    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394464    6014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394467    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394470    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394473    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394476    6014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394478    6014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:21.405493 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394481    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394483    6014 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394486    6014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394488    6014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394490    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394493    6014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394495    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394498    6014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394500    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394504    6014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394507    6014 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394511    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394514    6014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394516    6014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394519    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394521    6014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394526    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394529    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394531    6014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:21.405991 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394534    6014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394538    6014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394541    6014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394543    6014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394546    6014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394548    6014 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394551    6014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394555    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394558    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394562    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394564    6014 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394567    6014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394569    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394572    6014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394574    6014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.394599    6014 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:39:21.406561 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394666    6014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394671    6014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394675    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394678    6014 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394680    6014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394683    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394686    6014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394689    6014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394692    6014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394695    6014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394698    6014 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394700    6014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394703    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394705    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394709    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394711    6014 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394714    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394716    6014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394719    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394721    6014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:21.407150 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394724    6014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394726    6014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394729    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394731    6014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394734    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394739    6014 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394741    6014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394743    6014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394746    6014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394748    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394751    6014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394754    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394756    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394759    6014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394761    6014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394764    6014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394766    6014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394768    6014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394771    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:21.408509 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394773    6014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394776    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394778    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394781    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394784    6014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394786    6014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394789    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394793    6014 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394795    6014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394798    6014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394800    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394803    6014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394805    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394808    6014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394810    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394815    6014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394819    6014 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394823    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394828    6014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:21.409014 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394831    6014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394833    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394835    6014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394838    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394840    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394843    6014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394846    6014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394849    6014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394853    6014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394856    6014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394858    6014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394861    6014 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394863    6014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394866    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394868    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394871    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394873    6014 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394876    6014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394878    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:21.409511 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394881    6014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394885    6014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394888    6014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394890    6014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394893    6014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394895    6014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394898    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394900    6014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394903    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394905    6014 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394908    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394910    6014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394914    6014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394919    6014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394923    6014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394927    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394932    6014 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394936    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394940    6014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394944    6014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:21.409987 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394948    6014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394952    6014 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394955    6014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394957    6014 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394960    6014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394963    6014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394965    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394968    6014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.394970    6014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.398779    6014 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.398795    6014 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.398814    6014 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.398818    6014 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399155    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399176    6014 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399189    6014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399193    6014 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399198    6014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399202    6014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399206    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:21.410555 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399211    6014 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399215    6014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399219    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399223    6014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399228    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399232    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399238    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399242    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399251    6014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399255    6014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399260    6014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399264    6014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399268    6014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399273    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399277    6014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399281    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399285    6014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399289    6014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399293    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399297    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:21.411046 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399302    6014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399311    6014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399315    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399320    6014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399324    6014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399328    6014 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399341    6014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399346    6014 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399350    6014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399354    6014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399358    6014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399362    6014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399369    6014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399382    6014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399386    6014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399392    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399397    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399402    6014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399406    6014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:21.411560 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399410    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399414    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399419    6014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399423    6014 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399428    6014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399432    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399436    6014 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399446    6014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399450    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399454    6014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399458    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399462    6014 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399466    6014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399471    6014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399475    6014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399479    6014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399484    6014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399489    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399494    6014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399497    6014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:21.412039 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399507    6014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399512    6014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399516    6014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399520    6014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399524    6014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399528    6014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399533    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399537    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399541    6014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399545    6014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399552    6014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399557    6014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399568    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399572    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399577    6014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399582    6014 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399586    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399590    6014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399595    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:21.412554 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399599    6014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399603    6014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399607    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399612    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399616    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399625    6014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399629    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399634    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399638    6014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399643    6014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399648    6014 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399652    6014 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399656    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399660    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399666    6014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399670    6014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399675    6014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399678    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399687    6014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399691    6014 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:21.413028 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.399695    6014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.399750    6014 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400239    6014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400253    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400258    6014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400263    6014 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400267    6014 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400271    6014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400274    6014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400276    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400279    6014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400282    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400285    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400288    6014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400291    6014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400293    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:21.413545 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400296    6014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400298    6014 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400301    6014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400303    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400306    6014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400308    6014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400311    6014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400313    6014 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400315    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400318    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400321    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400324    6014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400327    6014 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400329    6014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400332    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400335    6014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400338    6014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400340    6014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400343    6014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400345    6014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:21.413934 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400348    6014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400350    6014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400353    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400355    6014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400358    6014 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400362    6014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400366    6014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400369    6014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400372    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400375    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400378    6014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400381    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400383    6014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400386    6014 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400388    6014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400391    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400393    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400396    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400398    6014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:21.414525 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400401    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400403    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400406    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400408    6014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400411    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400413    6014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400416    6014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400418    6014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400421    6014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400424    6014 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400426    6014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400429    6014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400431    6014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400433    6014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400436    6014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400439    6014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400441    6014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400444    6014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400446    6014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:21.415009 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400448    6014 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400451    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400454    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400457    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400460    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400463    6014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400465    6014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400468    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400470    6014 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400473    6014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400475    6014 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400478    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400480    6014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400483    6014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400485    6014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400488    6014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400490    6014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400493    6014 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400496    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400498    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:21.415599 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400501    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400504    6014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400506    6014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400509    6014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400511    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400514    6014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400516    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400519    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400521    6014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400523    6014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400526    6014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400528    6014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400531    6014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400533    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.400566    6014 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:39:21.416192 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400601    6014 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400605    6014 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400608    6014 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400611    6014 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400614    6014 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400617    6014 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400620    6014 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400623    6014 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400626    6014 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400628    6014 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400631    6014 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400634    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400636    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400639    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400641    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400644    6014 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400646    6014 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400649    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400651    6014 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400654    6014 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:21.416567 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400656    6014 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400658    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400661    6014 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400663    6014 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400666    6014 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400668    6014 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400671    6014 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400673    6014 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400675    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400678    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400680    6014 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400683    6014 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400685    6014 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400688    6014 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400691    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400693    6014 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400696    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400699    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400703    6014 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:21.417121 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400707    6014 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400711    6014 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400714    6014 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400719    6014 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400723    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400726    6014 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400729    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400733    6014 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400737    6014 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400740    6014 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400742    6014 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400745    6014 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400747    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400750    6014 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400752    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400756    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400760    6014 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400764    6014 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400768    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:21.417645 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400772    6014 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400775    6014 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400777    6014 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400780    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400783    6014 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400786    6014 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400791    6014 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400795    6014 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400798    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400805    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400809    6014 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400813    6014 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400816    6014 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400818    6014 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400821    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400824    6014 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400826    6014 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400829    6014 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400831    6014 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400834    6014 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:21.418305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400836    6014 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400839    6014 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400841    6014 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400844    6014 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400846    6014 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400850    6014 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400853    6014 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400856    6014 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400859    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400863    6014 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400865    6014 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400868    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400870    6014 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400872    6014 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400875    6014 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400880    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400883    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400885    6014 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400888    6014 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:21.419120 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400891    6014 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400893    6014 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400896    6014 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400899    6014 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400901    6014 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400904    6014 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400906    6014 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400909    6014 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:21.400911    6014 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.400966    6014 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.401927    6014 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.401974    6014 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.402158    6014 server.go:1014] "Starting client certificate rotation"
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.402254    6014 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.402304    6014 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:33:09 +0000 UTC" deadline="2028-06-23 23:46:11.067020981 +0000 UTC"
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.402331    6014 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14453h6m49.664691745s"
Oct 30 18:39:21.419778 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.402608    6014 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:39:21.420302 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.403722    6014 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:39:21.420302 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.405044    6014 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:39:21.420302 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.407099    6014 log.go:25] "Validated CRI v1 image API"
Oct 30 18:39:21.420302 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.407923    6014 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:39:21.420302 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.410305    6014 fs.go:135] Filesystem UUIDs: map[130e84bc-1515-41f0-bf18-284563424445:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a3db4ba5-e65d-4323-a9d1-a2d700002524:/dev/nvme0n1p3]
Oct 30 18:39:21.420466 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.410324    6014 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02aac1802c76057fad5ba627d7cd45c9b14cb558382cbced4aca6ab3663abd8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02aac1802c76057fad5ba627d7cd45c9b14cb558382cbced4aca6ab3663abd8e/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/194f6900f57730baa032582caa26db939ed29174446349177f142f0bb636b8f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/194f6900f57730baa032582caa26db939ed29174446349177f142f0bb636b8f9/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19e8bbf907ef6a3653fb434d65382689fae6823d7043a464362050ee0028f4c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19e8bbf907ef6a3653fb434d65382689fae6823d7043a464362050ee0028f4c1/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25139100cca5312773f20fc48d48fd08d0bd3c95aff49b9774511088baa3b43b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25139100cca5312773f20fc48d48fd08d0bd3c95aff49b9774511088baa3b43b/userdata/shm major:0 minor:255 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2de8db5c3bbc09064ad9052704fc421a8149e9c5a794d5148d74941f4ec1ea47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2de8db5c3bbc09064ad9052704fc421a8149e9c5a794d5148d74941f4ec1ea47/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e5aca942a63fe3102bf2f2c2b6bad1e47166e81b0f9547008ba4a8e6ad32ac5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e5aca942a63fe3102bf2f2c2b6bad1e47166e81b0f9547008ba4a8e6ad32ac5/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f5f6e6a669926fd233d4e892d2014812bcb0dea5a09c85127946930f19648be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f5f6e6a669926fd233d4e892d2014812bcb0dea5a09c85127946930f19648be/userdata/shm major:0 minor:518 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30023e8fbff341b10235f3e2091641296d825c74991ff9edf6568ad8934352b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30023e8fbff341b10235f3e2091641296d825c74991ff9edf6568ad8934352b7/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30ea2ad9a892d2d7ff1080af124641c146d08ae1cc4cba0ecad051080b21e84c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30ea2ad9a892d2d7ff1080af124641c146d08ae1cc4cba0ecad051080b21e84c/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3309f32908643a7ebc30a95f6d73c02dfdfe2734a33d056980f0e964cdf091b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3309f32908643a7ebc30a95f6d73c02dfdfe2734a33d056980f0e964cdf091b4/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/333698a8e60fe161576bb7e574f396a93f76b1f7440a50376c71ad6df33db7e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/333698a8e60fe161576bb7e574f396a93f76b1f7440a50376c71ad6df33db7e3/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ab2b76243302770f606dbe9948645a4929f6281387f5e93e8ea3742609236dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ab2b76243302770f606dbe9948645a4929f6281387f5e93e8ea3742609236dc/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43ad8896fedb69b24c0b82cec0a408554d789a07c371cb6b4fefae7c658c4d6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43ad8896fedb69b24c0b82cec0a408554d789a07c371cb6b4fefae7c658c4d6f/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53c7757eaac07f8ff12d7f43254531484e914baefabf34d7db6d7eb0e5b28e36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53c7757eaac07f8ff12d7f43254531484e914baefabf34d7db6d7eb0e5b28e36/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/597b9c55fb105ae5a89a46e99b6762bf9460c99371f2797b98afcc1442921283/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/597b9c55fb105ae5a89a46e99b6762bf9460c99371f2797b98afcc1442921283/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6eee339ba2566a84033aef619ba3119c5f5388d7af8d24b1e9fe2eaa14e9bcde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6eee339ba2566a84033aef619ba3119c5f5388d7af8d24b1e9fe2eaa14e9bcde/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77b2e7f0cab9f9b65467f7a928c2a720a6a7c5842410a587abeae62562f77e41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77b2e7f0cab9f9b65467f7a928c2a720a6a7c5842410a587abeae62562f77e41/userdata/shm major:0 minor:514 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80f03a6c2a94341aabe2204002e5b372c379b9f548b9d7e725600f186df1106c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80f03a6c2a94341aabe2204002e5b372c379b9f548b9d7e725600f186df1106c/userdata/shm major:0 minor:286 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f0941b9270fcdcf0be5202685ed7f97e1a19a72fb544148ab8ce523fad9437f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f0941b9270fcdcf0be5202685ed7f97e1a19a72fb544148ab8ce523fad9437f/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/905a8c6384e5e7c3a1c0fd3d0cde3432a320af6d22d82b0d7a9b72bc7edc4614/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/905a8c6384e5e7c3a1c0fd3d0cde3432a320af6d22d82b0d7a9b72bc7edc4614/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a557ca466124a05b24eac067708fabba82e0fb74c3cbed4d5737a4d43e9edf07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a557ca466124a05b24eac067708fabba82e0fb74c3cbed4d5737a4d43e9edf07/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5e91197db0c9ccd0a4eb969f4efe951a65f394cf376dd0af7e0bc9cf369a0a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5e91197db0c9ccd0a4eb969f4efe951a65f394cf376dd0af7e0bc9cf369a0a5/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c137d5e1976d9fa3d8d22f7da32ff85d3f077cee7d00165161516afd1e0b1598/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c137d5e1976d9fa3d8d22f7da32ff85d3f077cee7d00165161516afd1e0b1598/userdata/shm major:0 minor:516 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce815c96b5f3d14b0ca6eaea2f90f65a441db6cf7634b899d735dce29ab8a452/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce815c96b5f3d14b0ca6eaea2f90f65a441db6cf7634b899d735dce29ab8a452/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf8f58ce264a062b7151255601a6f5aab15ef00cc42e399e302fa0d279e113f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf8f58ce264a062b7151255601a6f5aab15ef00cc42e399e302fa0d279e113f9/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe32fd6bee74600dc227871e74853cd90c58569364d9b319f3341099fe742d65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe32fd6bee74600dc227871e74853cd90c58569364d9b319f3341099fe742d65/userdata/shm major:0 minor:469 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/028108f5-f006-42d0-9cfd-56f9b55e18ff/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/028108f5-f006-42d0-9cfd-56f9b55e18ff/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/028108f5-f006-42d0-9cfd-56f9b55e18ff/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/028108f5-f006-42d0-9cfd-56f9b55e18ff/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/028108f5-f006-42d0-9cfd-56f9b55e18ff/volumes/kubernetes.io~projected/kube-api-access-d2r6v:{mountpoint:/var/lib/kubelet/pods/028108f5-f006-42d0-9cfd-56f9b55e18ff/volumes/kubernetes.io~projected/kube-api-access-d2r6v major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b629f13-9f3f-41a9-9122-5d9fbe050beb/volumes/kubernetes.io~projected/kube-api-access-vvfqq:{mountpoint:/var/lib/kubelet/pods/0b629f13-9f3f-41a9-9122-5d9fbe050beb/volumes/kubernetes.io~projected/kube-api-access-vvfqq major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b629f13-9f3f-41a9-9122-5d9fbe050beb/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0b629f13-9f3f-41a9-9122-5d9fbe050beb/volumes/kubernetes.io~secret/cert major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1395368e-c046-4067-809a-6310f8d3c93d/volumes/kubernetes.io~projected/kube-api-access-fbtd6:{mountpoint:/var/lib/kubelet/pods/1395368e-c046-4067-809a-6310f8d3c93d/volumes/kubernetes.io~projected/kube-api-access-fbtd6 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~projected/kube-api-access-cjkqx:{mountpoint:/var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~projected/kube-api-access-cjkqx major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c93bbd-c046-4a50-83f2-ae8de8b64104/volumes/kubernetes.io~projected/kube-api-access-gj64r:{mountpoint:/var/lib/kubelet/pods/45c93bbd-c046-4a50-83f2-ae8de8b64104/volumes/kubernetes.io~projected/kube-api-access-gj64r major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d70a762-bdea-4394-8a95-5e5f0fb4042c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4d70a762-bdea-4394-8a95-5e5f0fb4042c/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/594ac1de-c6f2-4031-807e-08509c8a162d/volumes/kubernetes.io~projected/kube-api-access-db2ck:{mountpoint:/var/lib/kubelet/pods/594ac1de-c6f2-4031-807e-08509c8a162d/volumes/kubernetes.io~projected/kube-api-access-db2ck major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d438d4b-2388-4a65-a199-8cb4b3d27dc2/volumes/kubernetes.io~projected/kube-api-access-b6vzs:{mountpoint:/var/lib/kubelet/pods/5d438d4b-2388-4a65-a199-8cb4b3d27dc2/volumes/kubernetes.io~projected/kube-api-access-b6vzs major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d438d4b-2388-4a65-a199-8cb4b3d27dc2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5d438d4b-2388-4a65-a199-8cb4b3d27dc2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d438d4b-2388-4a65-a199-8cb4b3d27dc2/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/5d438d4b-2388-4a65-a199-8cb4b3d27dc2/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cebbc4c-d198-4222-968d-4ab6d035ee51/volumes/kubernetes.io~projected/kube-api-access-9vd7v:{mountpoint:/var/lib/kubelet/pods/6cebbc4c-d198-4222-968d-4ab6d035ee51/volumes/kubernetes.io~projected/kube-api-access-9vd7v major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cebbc4c-d198-4222-968d-4ab6d035ee51/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/6cebbc4c-d198-4222-968d-4ab6d035ee51/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cebbc4c-d198-4222-968d-4ab6d035ee51/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/6cebbc4c-d198-4222-968d-4ab6d035ee51/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cebbc4c-d198-4222-968d-4ab6d035ee51/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/6cebbc4c-d198-4222-968d-4ab6d035ee51/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/717a2238-075b-4288-9c74-1a306efb331d/volumes/kubernetes.io~projected/kube-api-access-xhfj5:{mountpoint:/var/lib/kubelet/pods/717a2238-075b-4288-9c74-1a306efb331d/volumes/kubernetes.io~projected/kube-api-access-xhfj5 major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/717a2238-075b-4288-9c74-1a306efb331d/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/717a2238-075b-4288-9c74-1a306efb331d/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/717a2238-075b-4288-9c74-1a306efb331d/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/717a2238-075b-4288-9c74-1a306efb331d/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73dec725-685b-4a6d-81b4-783f6c79494e/volumes/kubernetes.io~projected/kube-api-access-7lpn9:{mountpoint:/var/lib/kubelet/pods/73dec725-685b-4a6d-81b4-783f6c79494e/volumes/kubernetes.io~projected/kube-api-access-7lpn9 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73dec725-685b-4a6d-81b4-783f6c79494e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/73dec725-685b-4a6d-81b4-783f6c79494e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73dec725-685b-4a6d-81b4-783f6c79494e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/73dec725-685b-4a6d-81b4-783f6c79494e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/761ddebb-9140-4814-8145-3bb12ce8a148/volumes/kubernetes.io~projected/kube-api-access-4r8lg:{mountpoint:/var/lib/kubelet/pods/761ddebb-9140-4814-8145-3bb12ce8a148/volumes/kubernetes.io~projected/kube-api-access-4r8lg major:0 minor:513 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/909bd6eb-4140-4939-a5c1-4d148b3791dc/volumes/kubernetes.io~projected/kube-api-access-qg7ts:{mountpoint:/var/lib/kubelet/pods/909bd6eb-4140-4939-a5c1-4d148b3791dc/volumes/kubernetes.io~projected/kube-api-access-qg7ts major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/909bd6eb-4140-4939-a5c1-4d148b3791dc/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/909bd6eb-4140-4939-a5c1-4d148b3791dc/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1ed51f8-138a-4e90-bcfe-00cae5eba235/volumes/kubernetes.io~projected/kube-api-access-gkl46:{mountpoint:/var/lib/kubelet/pods/a1ed51f8-138a-4e90-bcfe-00cae5eba235/volumes/kubernetes.io~projected/kube-api-access-gkl46 major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1ed51f8-138a-4e90-bcfe-00cae5eba235/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a1ed51f8-138a-4e90-bcfe-00cae5eba235/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1ed51f8-138a-4e90-bcfe-00cae5eba235/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a1ed51f8-138a-4e90-bcfe-00cae5eba235/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~projected/kube-api-access-5xs2k:{mountpoint:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~projected/kube-api-access-5xs2k major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2168bb-3bf1-48ed-993d-4d0a014f00ae/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ab2168bb-3bf1-48ed-993d-4d0a014f00ae/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:508 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35/volumes/kubernetes.io~projected/kube-api-access-qhvz6:{mountpoint:/var/lib/kubelet/pods/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35/volumes/kubernetes.io~projected/kube-api-access-qhvz6 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2/volumes/kubernetes.io~projected/kube-api-access-7vgcm:{mountpoint:/var/lib/kubelet/pods/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2/volumes/kubernetes.io~projected/kube-api-access-7vgcm major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2e259e9-ce5d-458d-8b2f-7a5e4448e587/volumes/kubernetes.io~projected/kube-api-access-862sw:{mountpoint:/var/lib/kubelet/pods/b2e259e9-ce5d-458d-8b2f-7a5e4448e587/volumes/kubernetes.io~projected/kube-api-access-862sw major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdf7720c-65a8-494e-a679-239f3a59ba3f/volumes/kubernetes.io~projected/kube-api-access-5hqnx:{mountpoint:/var/lib/kubelet/pods/bdf7720c-65a8-494e-a679-239f3a59ba3f/volumes/kubernetes.io~projected/kube-api-access-5hqnx major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdf7720c-65a8-494e-a679-239f3a59ba3f/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/bdf7720c-65a8-494e-a679-239f3a59ba3f/volumes/kubernetes.io~secret/metrics-tls major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volume-subpaths/web-config/config-reloader/2 major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volume-subpaths/web-config/prometheus/15 major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~empty-dir/config-out major:0 minor:452 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~projected/kube-api-access-qtzzk:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~projected/kube-api-access-qtzzk major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~projected/tls-assets major:0 minor:451 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/config major:0 minor:458 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/web-config major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volume-subpaths/web-config/alertmanager/9 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volume-subpaths/web-config/config-reloader/6 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~empty-dir/config-out major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~projected/kube-api-access-bf5zd:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~projected/kube-api-access-bf5zd major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~projected/tls-assets major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/config-volume major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/web-config major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4/volumes/kubernetes.io~projected/kube-api-access-crdr9:{mountpoint:/var/lib/kubelet/pods/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4/volumes/kubernetes.io~projected/kube-api-access-crdr9 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4/volumes/kubernetes.io~secret/metrics-certs major:0 minor:512 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f478498d-4adf-441d-9dc6-1988d7c9e050/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f478498d-4adf-441d-9dc6-1988d7c9e050/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f478498d-4adf-441d-9dc6-1988d7c9e050/volumes/kubernetes.io~projected/kube-api-access-jzx9l:{mountpoint:/var/lib/kubelet/pods/f478498d-4adf-441d-9dc6-1988d7c9e050/volumes/kubernetes.io~projected/kube-api-access-jzx9l major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f478498d-4adf-441d-9dc6-1988d7c9e050/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f478498d-4adf-441d-9dc6-1988d7c9e050/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f95bb75f-1760-4132-bf36-20c2027b3059/volumes/kubernetes.io~projected/kube-api-access-qjnqm:{mountpoint:/var/lib/kubelet/pods/f95bb75f-1760-4132-bf36-20c2027b3059/volumes/kubernetes.io~projected/kube-api-access-qjnqm major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/afc06a4cbd7defd24c9e4c0154e16e5e0dc03102c3642535211cea6c9cf0ba60/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/16e70aadfebe0bfbdc9b5311076d863051465cdf9d359c46a4431d355e039def/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8f238c44472083fdbf7f6ad06f39c7f893c8b336594def9d495e9368f50e6ff2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/f8d5132da1e6273c8099670d4ef09c0144647b4e212a62f32632e69a5ee7a4a8/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/980db787d816308c9de1868c100394776d7f1c8330d10b54516032e992fde16e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/760f0c70421c91a5f01566ae11a6e153a6e564432fb7a0ae34ce068bbd1388c9/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/2b6d77a57400398419f0a40552d02117860f854ed8940f77d8c708d5fd503524/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/07b2f130d368db8b21ff9f7cb6781853e7d2a158bab7529d750c7406b7f32ab4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/c9b389cd0da90b2fd13c17d0f8b8f89810bfd72d1f3f82b8205e40176c72ef91/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/9fc852e089e7cbfb4d55ea896eb5041d205b573b0ec3c6a1bd592e45f7dcfe18/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/130f2af59bf3ff327f74a50eca248e7688c2ccdb49d904c866a0525d0cc6955f/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f26502ea0cdbadfca1545fd9577bbb7e46222be5b9e9f45c825892a6f830223b/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f3e18d16064789519b05bc1ccafd07e5b1ff49580785666c14901b64ff97fd5e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/a18792c34947779166614c2618dec5e55ac6118dd86286b2e31a182d8d8268f5/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/985c4f400a8a7b5fcd556a66e0f13d67ae36e008763d55b7cc40970609f81201/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/52f707440fc622452500467d63903add60bec090503224b4b79243a40b4c30cb/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/588aeb6235d8be092475d5699016e7f883b3503edfacbf2732851adaef2482fa/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/46d6a355458c1135b343b043edcd491faa88095398ff442779940008f1573771/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4765d09a654a7b90c36393442addef2dbcaf296676897b71160b887e231f0cc6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/14f2c50d2c78d03377230aa2bd0c3f6af1b19ece0d3c2050a3d0c4c65f2aac53/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/935c923e7a6a06a3d1ac0721a5ad738856f26944e197a25c9bd702866d4ceef6/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/25d91268594891b00734dbdffaeed15d851e92496cf721b2677ce37037564f9e/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/f7b97bd752c7a19e5f14b4cf9f43d9e9217e25fe581d17fb526d3350791e3710/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/790aaeb490b9fa9e61a2734cc9355b4f622d13c28a042cd5a7eb7a6e2c709dc2/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/57ea227f23a4c58631652796ca08896649a45502ac82c22883288d8c63fe37e0/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/f9e913840e171253e215518a4286138eda1ba7068147462a38e8a9ee6e8d5ccf/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/31627432b5fdba0db821ee0ee6b1ab6be93b6b23190700455110b64996a55cc8/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/824b70172639537a0a4c862030b3f675b62133ac1e5890ac1a37c6c6040f0026/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5a1adffd60ee228201eaec0cb9643a041cb19930f4d41c362110a7906c6770cc/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/6d8ac42b9ff60979b4c4dbd0a076fb8ca1f2e86cc2e2269100a0aea0028dca88/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/8d4a4db609a207bed5cfd8d42163c2ca3d880c6c89b31c21306ba8fe2660a5eb/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/9e9fc2ad588ddab01bb859560ae9f48b69527322c2da46d8df027ab3c386ce5d/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/9d66b233e9060ff1cca85c03dd6a69d3ba398938ac9ed990709ceb14fc702e96/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/eb3640247477cea14c0eeec8d7c3a021f3b1b67b48da9dd3aca639724db939eb/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/f008bbfecf6fb609f218c6eceadb43812df48182d178ab6d9d768a68a050a1c9/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/7b3572367904fefc3b35bcc4a7c2c2f38aef730c90b5ac79b74f23a71da8fc7f/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/02fc250fc63ac47100e56bfe53a499ed758537a97297f690c6356b6b803704b2/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/5460c26e5fed32842673fb87fc8a9939fc6385e6710a12ea83a0c3499b62f642/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/81620b11addc490aaab81c551c77e5d1003ceb1681f31694ae70942b1206f436/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/23038340da9f2c80d1cae73c6cf3a0562e6b649ab422e0d73e67a3cce028b4d7/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/7c90db8ef4eadab3ac5a57020f78ac809161efa692bc04e34492098b6ddffbd2/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/40ac800e50ee24cd497e88974b81cd4605cffb8f06c6c6e3190305ba89e453c8/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/cc28aafed4921f2ab3fbe7cb39cd35b27b4dafea857109c503da89969cedf50b/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/74aa4b37ed09d90e2c76a97d3bbfd41cde01ea7499534e3f232f8b4307fe5b23/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/fed56cc8eb03e9053d8ecf1aa5328a9ba8ffc30460b8b16632a339fd617d9334/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/e989b3da6e777057eca407643b11a67fcf2ebc089407594c8b8ac0e481bbfee7/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/21bc3848418a3d779be57a10567d41767fab3e0026579921189ef6f60a7b84ba/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/d376c4fbbc1ebfa3271f4b5a03dc1535f95eae09282922179558f37d7e08f2e6/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/a5371cdc905fce8f8a7c561bbcda57eb7f385854e3c20cdcc7a9cebffcf57f61/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/ac4e3fa237e16c9cb5b8e2748ccd761881bdb2f473370bcc61c6d4fe2173fd38/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/0e54658477d13ca6ef727c2d292fc62616d65ba4ff458aebe6223b58a6fda78a/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/1f7a988c7e3b8b89aa0b4cdd2f4650757f9aa7896ccd8cdb34d3b50232723370/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/9c02a0b6a60886df9bb12a5506b612221b9868400eb0b057d1213a3a6d585427/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/b13c790e91ef7c205eb35809f8f7c282bb24918ab5aafc238dfd5c96aac221b4/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/7a6cc39b994e360d1aec13b00ec257550dff6acf671a6a16273bf7df75127e0b/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/d1d888a11e76ad18fde8a3aece5da99c570ec03417d7ec462530f1629f00f294/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/cfc481e89b638f15266f841eaefa67c08cbf9d2346f4d42aa6d6465451256e0c/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/09aa3e62462957f3164bc7367bdd584703f079e478c51e73989b81578d3c899d/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/81fb10802afb41588006feb48c4b0f8a7371fdca17808fd98f99d2e2c3d71e3e/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/6fb01244691ed7689d266567d749f087d822dfa58db73e1fc627bd1f2b25ae5f/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/21dc509a382b73d714407d1bcc673250f054a79c3621d023eee66d1f10606139/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/017c586a043dca2b7076c8b9997a0579e54ebddddb88f30a085b52c1e3f78b98/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-459:{mountpoint:/var/lib/containers/storage/overlay/e149d9909c98f608162890a13eb939f831fc564f802b7bcf84f161fff8d935ae/merged major:0 minor:459 fsType:overlay blockSize:0} overlay_0-461:{mountpoint:/var/lib/containers/storage/overlay/b130d5774d90923398af1c972007f85411e1036c65290bb6a688d252e669753c/merged major:0 minor:461 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/57ce4d27b05750c8a83d3ec64bb78ef7414fb4331016fe4638be610bece6000c/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-475:{mountpoint:/var/lib/containers/storage/overlay/3b795d3bf18cf178ede54bfa0644e933bab4369f4ed484fd8ca0febe01c2ddd7/merged major:0 minor:475 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/6484e7a9d94f249fdac010abd8dc22960ed0166d5feed245ae1fdf99accc8228/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/3f940d248d36b5819d5645ce57ca8d1f05c99e5977d95266433f55b90a45b96d/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/510202041ebe41be9553e023dca894838cd6d772e66effaa6491de02ac2d28d6/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/49af0c02e587a4604378119ed75009953b48417c34685228aaa422e6c3639eae/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-498:{mountpoint:/var/lib/containers/storage/overlay/9f3f6fc168e6aba5f7c131c2b9e7e17366a7b1a7af56aebc2237d7000dcafee7/merged major:0 minor:498 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/0791f5360c7fdb93899258e77cceaab3c3ce410d97ecc6e45f66e3a918963f60/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-520:{mountpoint:/var/lib/containers/storage/overlay/22587389a1e0cfa3ef041e100b57568d658a2834c0393126cb1a8026b9b580ab/merged major:0 minor:520 fsType:overlay blockSize:0} overlay_0-522:{mountpoint:/var/lib/containers/storage/overlay/bee9f9467a3ac9419e59c74b32742c345122786acde03385c9cefe56e16fd359/merged major:0 minor:522 fsType:overlay blockSize:0} overlay_0-524:{mountpoint:/var/lib/containers/storage/overlay/504a44c2b67155682b07c18d4376ce34ec61d85accf4afb6928f2ad832ec5f75/merged major:0 minor:524 fsType:overlay blockSize:0} overlay_0-526:{mountpoint:/var/lib/containers/storage/overlay/fc632f7958af62397c9be8c42f39b1e32d594914991d2ffcd4a2a0a446f4222d/merged major:0 minor:526 fsType:overlay blockSize:0} overlay_0-528:{mountpoint:/var/lib/containers/storage/overlay/0742e359d82c473d0327c49267d840d1d20e22759dbc2477c07cc1781b55189d/merged major:0 minor:528 fsType:overlay blockSize:0} overlay_0-534:{mountpoint:/var/lib/containers/storage/overlay/dc998d39fd960ecb15e839e1000056e85c381c1e18e68071d13544d1c19b5777/merged major:0 minor:534 fsType:overlay blockSize:0} overlay_0-539:{mountpoint:/var/lib/containers/storage/overlay/13f5a5949981fb12b57fb4c05ebe71c7df8e233fac31905c845a1c98d290b5c2/merged major:0 minor:539 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/5b5b186add525a9626baa21b4721f50237fd39f19ced74e6bf6cda506de232a7/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/0dc17e5b22fe460f29a1784ef3ac0df6159486133d92c0ca2086dc372ff001d8/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f25453091ab6f6f72979b7c9cd927667cd365ca9f76c270ab4207728a16751df/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/b834cc7d02b2a060128b958fa33cb9c984e603c06fcd7cce7d4c88bcd6505d1a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/d240d577d4b3963a391d5dbfc721328ea24de3c9d0f8218d60b4930d30127290/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c3b9f62b0966d8d640196a8e4f989904486c7216d6d691e93ed4ea378d47b38f/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/9110b243957fe4d90e3ad9ae93d14362f2c2bbeb5e4d21c39fa49862cbe69411/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a1539ebbb5274a70ce107ce9bbe2dc5216be7383ace64ee604ce88439c29af50/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/695738ae5a445ea0aed8d4c64aa39434aa71a1db897e123d1be6a79a5cd7fdf3/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7d4a04178eb7360009da7d745f5071c011feb7e48236222d6b54e86a65f4494a/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 18:39:21.420768 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.419952    6014 manager.go:217] Machine: {Timestamp:2026-10-30 18:39:21.419038206 +0000 UTC m=+0.062205812 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101892 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b2460964a3bc81d80f8d4cc5ef51d SystemUUID:ec2b2460-964a-3bc8-1d80-f8d4cc5ef51d BootID:ac3f7d52-8e75-4e4f-b991-8398ebb2412b Filesystems:[{Device:/var/lib/kubelet/pods/028108f5-f006-42d0-9cfd-56f9b55e18ff/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/028108f5-f006-42d0-9cfd-56f9b55e18ff/volumes/kubernetes.io~projected/kube-api-access-d2r6v DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bdf7720c-65a8-494e-a679-239f3a59ba3f/volumes/kubernetes.io~projected/kube-api-access-5hqnx DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53c7757eaac07f8ff12d7f43254531484e914baefabf34d7db6d7eb0e5b28e36/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5d438d4b-2388-4a65-a199-8cb4b3d27dc2/volumes/kubernetes.io~projected/kube-api-access-b6vzs DeviceMajor:0 DeviceMinor:253 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:332 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4d70a762-bdea-4394-8a95-5e5f0fb4042c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:360 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:362 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:452 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-498 DeviceMajor:0 DeviceMinor:498 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-522 DeviceMajor:0 DeviceMinor:522 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-524 DeviceMajor:0 DeviceMinor:524 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4/volumes/kubernetes.io~projected/kube-api-access-crdr9 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f95bb75f-1760-4132-bf36-20c2027b3059/volumes/kubernetes.io~projected/kube-api-access-qjnqm DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:512 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/594ac1de-c6f2-4031-807e-08509c8a162d/volumes/kubernetes.io~projected/kube-api-access-db2ck DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:293 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-475 DeviceMajor:0 DeviceMinor:475 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-520 DeviceMajor:0 DeviceMinor:520 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f478498d-4adf-441d-9dc6-1988d7c9e050/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/717a2238-075b-4288-9c74-1a306efb331d/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a1ed51f8-138a-4e90-bcfe-00cae5eba235/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:246 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80f03a6c2a94341aabe2204002e5b372c379b9f548b9d7e725600f186df1106c/userdata/shm DeviceMajor:0 DeviceMinor:286 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73dec725-685b-4a6d-81b4-783f6c79494e/volumes/kubernetes.io~projected/kube-api-access-7lpn9 DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/905a8c6384e5e7c3a1c0fd3d0cde3432a320af6d22d82b0d7a9b72bc7edc4614/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e5aca942a63fe3102bf2f2c2b6bad1e47166e81b0f9547008ba4a8e6ad32ac5/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35/volumes/kubernetes.io~projected/kube-api-access-qhvz6 DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:458 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f478498d-4adf-441d-9dc6-1988d7c9e050/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:294 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:445 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-461 DeviceMajor:0 DeviceMinor:461 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f5f6e6a669926fd233d4e892d2014812bcb0dea5a09c85127946930f19648be/userdata/shm DeviceMajor:0 DeviceMinor:518 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f478498d-4adf-441d-9dc6-1988d7c9e050/volumes/kubernetes.io~projected/kube-api-access-jzx9l DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce815c96b5f3d14b0ca6eaea2f90f65a441db6cf7634b899d735dce29ab8a452/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73dec725-685b-4a6d-81b4-783f6c79494e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:298 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/909bd6eb-4140-4939-a5c1-4d148b3791dc/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:345 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:455 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-459 DeviceMajor:0 DeviceMinor:459 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe32fd6bee74600dc227871e74853cd90c58569364d9b319f3341099fe742d65/userdata/shm DeviceMajor:0 DeviceMinor:469 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5e91197db0c9ccd0a4eb969f4efe951a65f394cf376dd0af7e0bc9cf369a0a5/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:450 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-528 DeviceMajor:0 DeviceMinor:528 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02aac1802c76057fad5ba627d7cd45c9b14cb558382cbced4aca6ab3663abd8e/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6eee339ba2566a84033aef619ba3119c5f5388d7af8d24b1e9fe2eaa14e9bcde/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a1ed51f8-138a-4e90-bcfe-00cae5eba235/volumes/kubernetes.io~projected/kube-api-access-gkl46 DeviceMajor:0 DeviceMinor:254 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:299 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6cebbc4c-d198-4222-968d-4ab6d035ee51/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:347 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6cebbc4c-d198-4222-968d-4ab6d035ee51/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:348 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2e259e9-ce5d-458d-8b2f-7a5e4448e587/volumes/kubernetes.io~projected/kube-api-access-862sw DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73dec725-685b-4a6d-81b4-783f6c79494e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25139100cca5312773f20fc48d48fd08d0bd3c95aff49b9774511088baa3b43b/userdata/shm DeviceMajor:0 DeviceMinor:255 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~projected/kube-api-access-5xs2k DeviceMajor:0 DeviceMinor:336 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:451 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1ed51f8-138a-4e90-bcfe-00cae5eba235/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:245 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:453 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/761ddebb-9140-4814-8145-3bb12ce8a148/volumes/kubernetes.io~projected/kube-api-access-4r8lg DeviceMajor:0 DeviceMinor:513 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-534 DeviceMajor:0 DeviceMinor:534 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d438d4b-2388-4a65-a199-8cb4b3d27dc2/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:247 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~projected/kube-api-access-cjkqx DeviceMajor:0 DeviceMinor:363 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/597b9c55fb105ae5a89a46e99b6762bf9460c99371f2797b98afcc1442921283/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:294 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:449 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:454 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/028108f5-f006-42d0-9cfd-56f9b55e18ff/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30ea2ad9a892d2d7ff1080af124641c146d08ae1cc4cba0ecad051080b21e84c/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bdf7720c-65a8-494e-a679-239f3a59ba3f/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:196 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:296 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:457 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/909bd6eb-4140-4939-a5c1-4d148b3791dc/volumes/kubernetes.io~projected/kube-api-access-qg7ts DeviceMajor:0 DeviceMinor:346 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:297 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3309f32908643a7ebc30a95f6d73c02dfdfe2734a33d056980f0e964cdf091b4/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5d438d4b-2388-4a65-a199-8cb4b3d27dc2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:244 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/717a2238-075b-4288-9c74-1a306efb331d/volumes/kubernetes.io~projected/kube-api-access-xhfj5 DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:333 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6cebbc4c-d198-4222-968d-4ab6d035ee51/volumes/kubernetes.io~projected/kube-api-access-9vd7v DeviceMajor:0 DeviceMinor:352 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:453 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:453 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1395368e-c046-4067-809a-6310f8d3c93d/volumes/kubernetes.io~projected/kube-api-access-fbtd6 DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a557ca466124a05b24eac067708fabba82e0fb74c3cbed4d5737a4d43e9edf07/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45c93bbd-c046-4a50-83f2-ae8de8b64104/volumes/kubernetes.io~projected/kube-api-access-gj64r DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ab2b76243302770f606dbe9948645a4929f6281387f5e93e8ea3742609236dc/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/717a2238-075b-4288-9c74-1a306efb331d/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c137d5e1976d9fa3d8d22f7da32ff85d3f077cee7d00165161516afd1e0b1598/userdata/shm DeviceMajor:0 DeviceMinor:516 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2/volumes/kubernetes.io~projected/kube-api-access-7vgcm DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:456 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:292 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:331 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/194f6900f57730baa032582caa26db939ed29174446349177f142f0bb636b8f9/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2168bb-3bf1-48ed-993d-4d0a014f00ae/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:508 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77b2e7f0cab9f9b65467f7a928c2a720a6a7c5842410a587abeae62562f77e41/userdata/shm DeviceMajor:0 DeviceMinor:514 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf8f58ce264a062b7151255601a6f5aab15ef00cc42e399e302fa0d279e113f9/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:453 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-526 DeviceMajor:0 DeviceMinor:526 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f0941b9270fcdcf0be5202685ed7f97e1a19a72fb544148ab8ce523fad9437f/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes/kubernetes.io~projected/kube-api-access-bf5zd DeviceMajor:0 DeviceMinor:304 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6cebbc4c-d198-4222-968d-4ab6d035ee51/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:349 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes/kubernetes.io~projected/kube-api-access-qtzzk DeviceMajor:0 DeviceMinor:466 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-539 DeviceMajor:0 DeviceMinor:539 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30023e8fbff341b10235f3e2091641296d825c74991ff9edf6568ad8934352b7/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6cfb91c-11df-4d59-b723-320754d81b99/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b629f13-9f3f-41a9-9122-5d9fbe050beb/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:200 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:359 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2cf0ddaf-aa3c-4390-b911-cab27953af11/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:361 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43ad8896fedb69b24c0b82cec0a408554d789a07c371cb6b4fefae7c658c4d6f/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2de8db5c3bbc09064ad9052704fc421a8149e9c5a794d5148d74941f4ec1ea47/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/333698a8e60fe161576bb7e574f396a93f76b1f7440a50376c71ad6df33db7e3/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b629f13-9f3f-41a9-9122-5d9fbe050beb/volumes/kubernetes.io~projected/kube-api-access-vvfqq DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19e8bbf907ef6a3653fb434d65382689fae6823d7043a464362050ee0028f4c1/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02aac1802c76057 MacAddress:4a:b0:33:ba:8d:58 Speed:10000 Mtu:8901} {Name:194f6900f57730b MacAddress:ee:b8:2d:ca:d5:53 Speed:10000 Mtu:8901} {Name:19e8bbf907ef6a3 MacAddress:a6:0a:46:16:24:b7 Speed:10000 Mtu:8901} {Name:2f5f6e6a669926f MacAddress:ca:b8:6b:88:3b:75 Speed:10000 Mtu:8901} {Name:43ad8896fedb69b MacAddress:6a:09:42:0c:26:00 Speed:10000 Mtu:8901} {Name:53c7757eaac07f8 MacAddress:0e:5b:2a:59:89:6c Speed:10000 Mtu:8901} {Name:597b9c55fb105ae MacAddress:42:6f:3e:d4:15:dd Speed:10000 Mtu:8901} {Name:6eee339ba2566a8 MacAddress:7e:37:2c:6e:3b:43 Speed:10000 Mtu:8901} {Name:77b2e7f0cab9f9b MacAddress:02:8b:01:50:c0:d2 Speed:10000 Mtu:8901} {Name:80f03a6c2a94341 MacAddress:02:65:db:f1:1b:7f Speed:10000 Mtu:8901} {Name:905a8c6384e5e7c MacAddress:a6:68:74:4d:d1:48 Speed:10000 Mtu:8901} {Name:b5e91197db0c9cc MacAddress:f6:2d:1b:2f:30:6a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:66:a3:39:f8:cd Speed:0 Mtu:9001} {Name:br-int MacAddress:de:37:d3:0c:e9:fb Speed:0 Mtu:8901} {Name:c137d5e1976d9fa MacAddress:a6:34:3d:20:0e:21 Speed:10000 Mtu:8901} {Name:cf8f58ce264a062 MacAddress:be:aa:6e:5a:87:71 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:66:a3:39:f8:cd Speed:0 Mtu:9001} {Name:fe32fd6bee74600 MacAddress:fe:d4:fa:70:c1:81 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:b6:9f:91:a9:df:44 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5a:5a:ad:d3:30:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:39:21.420768 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.420444    6014 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:39:21.420768 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.420498    6014 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:39:21.421046 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.420799    6014 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:39:21.421046 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.420821    6014 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-97.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:39:21.421046 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.420946    6014 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:39:21.421046 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.420954    6014 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:39:21.421046 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.420964    6014 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:39:21.421046 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.420984    6014 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:39:21.421260 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.421082    6014 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:39:21.421288 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.421265    6014 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:39:21.421381 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.421370    6014 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:39:21.421413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.421408    6014 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:39:21.421446 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.421424    6014 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:39:21.421446 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.421434    6014 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:39:21.421446 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.421445    6014 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:39:21.421543 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.421453    6014 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:39:21.422459 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.422443    6014 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:39:21.422459 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.422460    6014 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:39:21.422982 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.422965    6014 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:39:21.425769 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.425755    6014 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:39:21.425823 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.425770    6014 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:39:21.425887 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.425878    6014 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 18:39:21.426085 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426074    6014 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:39:21.426118 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426103    6014 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:39:21.426244 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426233    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:39:21.426281 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426254    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:39:21.426281 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426261    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:39:21.426281 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426267    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:39:21.426281 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426274    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:39:21.426281 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426280    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:39:21.426413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426285    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:39:21.426413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426291    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:39:21.426413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426298    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:39:21.426413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426303    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:39:21.426413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426319    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:39:21.426413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426328    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:39:21.426413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426382    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:39:21.426413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.426390    6014 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:39:21.429772 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.429758    6014 server.go:1263] "Started kubelet"
Oct 30 18:39:21.429904 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.429852    6014 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:39:21.430393 ip-10-0-143-97 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:39:21.431800 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.430635    6014 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:39:21.431800 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.430745    6014 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:39:21.431800 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.431384    6014 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:39:21.432184 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.432111    6014 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:39:21.437080 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.437063    6014 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:39:21.437202 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.437087    6014 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.437234    6014 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:33:10 +0000 UTC" deadline="2028-04-20 10:07:39.372874721 +0000 UTC"
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.437258    6014 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12903h28m17.935621004s"
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.437298    6014 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.437308    6014 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:21.437299    6014 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-97.us-west-1.compute.internal\" not found"
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.437463    6014 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.439499    6014 factory.go:55] Registering systemd factory
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.439520    6014 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.439725    6014 factory.go:153] Registering CRI-O factory
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.439735    6014 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.439801    6014 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.439839    6014 factory.go:103] Registering Raw factory
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.439854    6014 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:39:21.440347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.440202    6014 manager.go:319] Starting recovery of all containers
Oct 30 18:39:21.441359 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.441312    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae5a6b4e-719e-4978-afa3-8d179bbcbdd2" volumeName="kubernetes.io/projected/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-kube-api-access-7vgcm" seLinuxMountContext=""
Oct 30 18:39:21.441440 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.441358    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 18:39:21.441440 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.441373    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4" volumeName="kubernetes.io/projected/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-kube-api-access-crdr9" seLinuxMountContext=""
Oct 30 18:39:21.441440 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.441386    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="028108f5-f006-42d0-9cfd-56f9b55e18ff" volumeName="kubernetes.io/empty-dir/028108f5-f006-42d0-9cfd-56f9b55e18ff-tmp" seLinuxMountContext=""
Oct 30 18:39:21.441440 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.441397    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c93bbd-c046-4a50-83f2-ae8de8b64104" volumeName="kubernetes.io/empty-dir/45c93bbd-c046-4a50-83f2-ae8de8b64104-tmp-dir" seLinuxMountContext=""
Oct 30 18:39:21.441440 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.441409    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d70a762-bdea-4394-8a95-5e5f0fb4042c" volumeName="kubernetes.io/secret/4d70a762-bdea-4394-8a95-5e5f0fb4042c-agent-certs" seLinuxMountContext=""
Oct 30 18:39:21.441440 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.441420    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6cfb91c-11df-4d59-b723-320754d81b99" volumeName="kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-grpc-tls" seLinuxMountContext=""
Oct 30 18:39:21.441440 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.441432    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f95bb75f-1760-4132-bf36-20c2027b3059" volumeName="kubernetes.io/projected/f95bb75f-1760-4132-bf36-20c2027b3059-kube-api-access-qjnqm" seLinuxMountContext=""
Oct 30 18:39:21.441808 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.441447    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1395368e-c046-4067-809a-6310f8d3c93d" volumeName="kubernetes.io/projected/1395368e-c046-4067-809a-6310f8d3c93d-kube-api-access-fbtd6" seLinuxMountContext=""
Oct 30 18:39:21.441808 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.441461    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cebbc4c-d198-4222-968d-4ab6d035ee51" volumeName="kubernetes.io/empty-dir/6cebbc4c-d198-4222-968d-4ab6d035ee51-audit-log" seLinuxMountContext=""
Oct 30 18:39:21.441808 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.441473    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 18:39:21.441808 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.441485    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f54b22-ebcb-43e3-8a45-d61242595fca" volumeName="kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:21.442030 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442009    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="028108f5-f006-42d0-9cfd-56f9b55e18ff" volumeName="kubernetes.io/empty-dir/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-tuned" seLinuxMountContext=""
Oct 30 18:39:21.442091 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442036    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ed51f8-138a-4e90-bcfe-00cae5eba235" volumeName="kubernetes.io/configmap/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 18:39:21.442091 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442059    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" volumeName="kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 18:39:21.442091 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442074    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-tls-assets" seLinuxMountContext=""
Oct 30 18:39:21.442272 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442089    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-grpc-tls" seLinuxMountContext=""
Oct 30 18:39:21.442272 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442107    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 18:39:21.442272 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442123    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ed51f8-138a-4e90-bcfe-00cae5eba235" volumeName="kubernetes.io/configmap/a1ed51f8-138a-4e90-bcfe-00cae5eba235-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:21.442272 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442165    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d438d4b-2388-4a65-a199-8cb4b3d27dc2" volumeName="kubernetes.io/empty-dir/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-textfile" seLinuxMountContext=""
Oct 30 18:39:21.442272 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442183    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cebbc4c-d198-4222-968d-4ab6d035ee51" volumeName="kubernetes.io/projected/6cebbc4c-d198-4222-968d-4ab6d035ee51-kube-api-access-9vd7v" seLinuxMountContext=""
Oct 30 18:39:21.442272 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442203    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 18:39:21.442272 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442225    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="717a2238-075b-4288-9c74-1a306efb331d" volumeName="kubernetes.io/secret/717a2238-075b-4288-9c74-1a306efb331d-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 18:39:21.442272 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442245    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73dec725-685b-4a6d-81b4-783f6c79494e" volumeName="kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert" seLinuxMountContext=""
Oct 30 18:39:21.442272 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442261    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdf7720c-65a8-494e-a679-239f3a59ba3f" volumeName="kubernetes.io/projected/bdf7720c-65a8-494e-a679-239f3a59ba3f-kube-api-access-5hqnx" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442283    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-config" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442303    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdf7720c-65a8-494e-a679-239f3a59ba3f" volumeName="kubernetes.io/secret/bdf7720c-65a8-494e-a679-239f3a59ba3f-metrics-tls" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442318    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-web-config" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442379    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1395368e-c046-4067-809a-6310f8d3c93d" volumeName="kubernetes.io/configmap/1395368e-c046-4067-809a-6310f8d3c93d-iptables-alerter-script" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442395    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d438d4b-2388-4a65-a199-8cb4b3d27dc2" volumeName="kubernetes.io/secret/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-tls" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442408    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac09d41c-10f4-4d5d-a8cb-1e638a70ab35" volumeName="kubernetes.io/configmap/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442425    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442436    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442448    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f54b22-ebcb-43e3-8a45-d61242595fca" volumeName="kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442459    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="594ac1de-c6f2-4031-807e-08509c8a162d" volumeName="kubernetes.io/configmap/594ac1de-c6f2-4031-807e-08509c8a162d-serviceca" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442469    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73dec725-685b-4a6d-81b4-783f6c79494e" volumeName="kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442480    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="909bd6eb-4140-4939-a5c1-4d148b3791dc" volumeName="kubernetes.io/projected/909bd6eb-4140-4939-a5c1-4d148b3791dc-kube-api-access-qg7ts" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442491    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442503    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442514    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f478498d-4adf-441d-9dc6-1988d7c9e050" volumeName="kubernetes.io/projected/f478498d-4adf-441d-9dc6-1988d7c9e050-kube-api-access-jzx9l" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442527    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73dec725-685b-4a6d-81b4-783f6c79494e" volumeName="kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442538    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8659e3f-b750-4d38-aa0b-7cc7f883f9b4" volumeName="kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442549    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f478498d-4adf-441d-9dc6-1988d7c9e050" volumeName="kubernetes.io/secret/f478498d-4adf-441d-9dc6-1988d7c9e050-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442561    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="717a2238-075b-4288-9c74-1a306efb331d" volumeName="kubernetes.io/configmap/717a2238-075b-4288-9c74-1a306efb331d-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442573    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ed51f8-138a-4e90-bcfe-00cae5eba235" volumeName="kubernetes.io/secret/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442583    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442594    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="909bd6eb-4140-4939-a5c1-4d148b3791dc" volumeName="kubernetes.io/secret/909bd6eb-4140-4939-a5c1-4d148b3791dc-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442605    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6cfb91c-11df-4d59-b723-320754d81b99" volumeName="kubernetes.io/projected/a6cfb91c-11df-4d59-b723-320754d81b99-kube-api-access-5xs2k" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442616    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2168bb-3bf1-48ed-993d-4d0a014f00ae" volumeName="kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442629    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442652    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="028108f5-f006-42d0-9cfd-56f9b55e18ff" volumeName="kubernetes.io/projected/028108f5-f006-42d0-9cfd-56f9b55e18ff-kube-api-access-d2r6v" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442663    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73dec725-685b-4a6d-81b4-783f6c79494e" volumeName="kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442674    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6cfb91c-11df-4d59-b723-320754d81b99" volumeName="kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442686    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6cfb91c-11df-4d59-b723-320754d81b99" volumeName="kubernetes.io/empty-dir/a6cfb91c-11df-4d59-b723-320754d81b99-tmp" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442698    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" volumeName="kubernetes.io/projected/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-kube-api-access-862sw" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442710    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdf7720c-65a8-494e-a679-239f3a59ba3f" volumeName="kubernetes.io/configmap/bdf7720c-65a8-494e-a679-239f3a59ba3f-config-volume" seLinuxMountContext=""
Oct 30 18:39:21.442714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442731    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cebbc4c-d198-4222-968d-4ab6d035ee51" volumeName="kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-secret-metrics-server-tls" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442741    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="717a2238-075b-4288-9c74-1a306efb331d" volumeName="kubernetes.io/projected/717a2238-075b-4288-9c74-1a306efb331d-kube-api-access-xhfj5" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442754    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73dec725-685b-4a6d-81b4-783f6c79494e" volumeName="kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442764    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac09d41c-10f4-4d5d-a8cb-1e638a70ab35" volumeName="kubernetes.io/secret/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442775    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442785    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f54b22-ebcb-43e3-8a45-d61242595fca" volumeName="kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442797    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cebbc4c-d198-4222-968d-4ab6d035ee51" volumeName="kubernetes.io/configmap/6cebbc4c-d198-4222-968d-4ab6d035ee51-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442807    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d438d4b-2388-4a65-a199-8cb4b3d27dc2" volumeName="kubernetes.io/configmap/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442830    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f54b22-ebcb-43e3-8a45-d61242595fca" volumeName="kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-web-config" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442842    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f478498d-4adf-441d-9dc6-1988d7c9e050" volumeName="kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442852    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" volumeName="kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442863    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f54b22-ebcb-43e3-8a45-d61242595fca" volumeName="kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442875    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f54b22-ebcb-43e3-8a45-d61242595fca" volumeName="kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-volume" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442885    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-kube-api-access-qtzzk" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442895    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6cfb91c-11df-4d59-b723-320754d81b99" volumeName="kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442906    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac09d41c-10f4-4d5d-a8cb-1e638a70ab35" volumeName="kubernetes.io/empty-dir/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-data-volume" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442915    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f54b22-ebcb-43e3-8a45-d61242595fca" volumeName="kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442926    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cf0ddaf-aa3c-4390-b911-cab27953af11" volumeName="kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442938    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d70a762-bdea-4394-8a95-5e5f0fb4042c" volumeName="kubernetes.io/configmap/4d70a762-bdea-4394-8a95-5e5f0fb4042c-konnectivity-ca" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442948    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442958    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f54b22-ebcb-43e3-8a45-d61242595fca" volumeName="kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-kube-api-access-bf5zd" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442970    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cf0ddaf-aa3c-4390-b911-cab27953af11" volumeName="kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-telemeter-client-tls" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442983    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d438d4b-2388-4a65-a199-8cb4b3d27dc2" volumeName="kubernetes.io/secret/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.442994    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cebbc4c-d198-4222-968d-4ab6d035ee51" volumeName="kubernetes.io/configmap/6cebbc4c-d198-4222-968d-4ab6d035ee51-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443007    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ed51f8-138a-4e90-bcfe-00cae5eba235" volumeName="kubernetes.io/projected/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-api-access-gkl46" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443018    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-config-out" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443029    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="594ac1de-c6f2-4031-807e-08509c8a162d" volumeName="kubernetes.io/projected/594ac1de-c6f2-4031-807e-08509c8a162d-kube-api-access-db2ck" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443041    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d438d4b-2388-4a65-a199-8cb4b3d27dc2" volumeName="kubernetes.io/projected/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-kube-api-access-b6vzs" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443052    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f478498d-4adf-441d-9dc6-1988d7c9e050" volumeName="kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-env-overrides" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443062    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cebbc4c-d198-4222-968d-4ab6d035ee51" volumeName="kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-client-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443074    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="717a2238-075b-4288-9c74-1a306efb331d" volumeName="kubernetes.io/secret/717a2238-075b-4288-9c74-1a306efb331d-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443085    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f478498d-4adf-441d-9dc6-1988d7c9e050" volumeName="kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-ovnkube-config" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443095    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443106    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f54b22-ebcb-43e3-8a45-d61242595fca" volumeName="kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-tls-assets" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443118    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b629f13-9f3f-41a9-9122-5d9fbe050beb" volumeName="kubernetes.io/projected/0b629f13-9f3f-41a9-9122-5d9fbe050beb-kube-api-access-vvfqq" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443131    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cf0ddaf-aa3c-4390-b911-cab27953af11" volumeName="kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443160    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cf0ddaf-aa3c-4390-b911-cab27953af11" volumeName="kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443170    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ed51f8-138a-4e90-bcfe-00cae5eba235" volumeName="kubernetes.io/empty-dir/a1ed51f8-138a-4e90-bcfe-00cae5eba235-volume-directive-shadow" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443180    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ed51f8-138a-4e90-bcfe-00cae5eba235" volumeName="kubernetes.io/secret/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443191    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b629f13-9f3f-41a9-9122-5d9fbe050beb" volumeName="kubernetes.io/secret/0b629f13-9f3f-41a9-9122-5d9fbe050beb-cert" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443205    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cf0ddaf-aa3c-4390-b911-cab27953af11" volumeName="kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-secret-telemeter-client" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443217    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac09d41c-10f4-4d5d-a8cb-1e638a70ab35" volumeName="kubernetes.io/projected/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-kube-api-access-qhvz6" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443230    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdf7720c-65a8-494e-a679-239f3a59ba3f" volumeName="kubernetes.io/empty-dir/bdf7720c-65a8-494e-a679-239f3a59ba3f-tmp-dir" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443240    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f54b22-ebcb-43e3-8a45-d61242595fca" volumeName="kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-out" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443251    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f54b22-ebcb-43e3-8a45-d61242595fca" volumeName="kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-main-db" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443263    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6cfb91c-11df-4d59-b723-320754d81b99" volumeName="kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443278    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f54b22-ebcb-43e3-8a45-d61242595fca" volumeName="kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443291    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6cfb91c-11df-4d59-b723-320754d81b99" volumeName="kubernetes.io/configmap/a6cfb91c-11df-4d59-b723-320754d81b99-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443302    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cf0ddaf-aa3c-4390-b911-cab27953af11" volumeName="kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443317    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d438d4b-2388-4a65-a199-8cb4b3d27dc2" volumeName="kubernetes.io/configmap/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443341    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73dec725-685b-4a6d-81b4-783f6c79494e" volumeName="kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443354    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="761ddebb-9140-4814-8145-3bb12ce8a148" volumeName="kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443365    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6cfb91c-11df-4d59-b723-320754d81b99" volumeName="kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443378    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6cfb91c-11df-4d59-b723-320754d81b99" volumeName="kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443411    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f95bb75f-1760-4132-bf36-20c2027b3059" volumeName="kubernetes.io/configmap/f95bb75f-1760-4132-bf36-20c2027b3059-multus-daemon-config" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443423    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cf0ddaf-aa3c-4390-b911-cab27953af11" volumeName="kubernetes.io/projected/2cf0ddaf-aa3c-4390-b911-cab27953af11-kube-api-access-cjkqx" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443468    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73dec725-685b-4a6d-81b4-783f6c79494e" volumeName="kubernetes.io/empty-dir/73dec725-685b-4a6d-81b4-783f6c79494e-tmp" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443480    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cebbc4c-d198-4222-968d-4ab6d035ee51" volumeName="kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443492    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73dec725-685b-4a6d-81b4-783f6c79494e" volumeName="kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443503    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" volumeName="kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443521    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d615363f-bfb2-4a9f-8678-774d3effd7d6" volumeName="kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443532    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f54b22-ebcb-43e3-8a45-d61242595fca" volumeName="kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-cluster-tls-config" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443542    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f95bb75f-1760-4132-bf36-20c2027b3059" volumeName="kubernetes.io/configmap/f95bb75f-1760-4132-bf36-20c2027b3059-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443553    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cf0ddaf-aa3c-4390-b911-cab27953af11" volumeName="kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-federate-client-tls" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443564    6014 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c93bbd-c046-4a50-83f2-ae8de8b64104" volumeName="kubernetes.io/projected/45c93bbd-c046-4a50-83f2-ae8de8b64104-kube-api-access-gj64r" seLinuxMountContext=""
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443575    6014 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:39:21.444259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.443583    6014 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:39:21.466532 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.466476    6014 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:39:21.467953 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.467930    6014 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:39:21.467953 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.467951    6014 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:39:21.468101 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.467972    6014 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:39:21.468101 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:21.468010    6014 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 18:39:21.480197 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.480161    6014 generic.go:356] "Generic (PLEG): container finished" podID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerID="004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced" exitCode=0
Oct 30 18:39:21.486587 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.486373    6014 generic.go:356] "Generic (PLEG): container finished" podID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerID="6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133" exitCode=0
Oct 30 18:39:21.522893 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.522838    6014 generic.go:356] "Generic (PLEG): container finished" podID="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" containerID="023b0b58425ce5b846445e1b4169ec231dacf047bba51dd8db786d011f2aed04" exitCode=0
Oct 30 18:39:21.522893 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.522880    6014 generic.go:356] "Generic (PLEG): container finished" podID="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" containerID="6b8d8243e4acfc605aed8b66e020f742f9fc8b931c03a1b79d858b3b0a453da4" exitCode=0
Oct 30 18:39:21.522893 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.522890    6014 generic.go:356] "Generic (PLEG): container finished" podID="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" containerID="4ede6998f73228457b56c4263df7e0f76a11652555e90546b3da65598b41b24f" exitCode=0
Oct 30 18:39:21.522893 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.522904    6014 generic.go:356] "Generic (PLEG): container finished" podID="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" containerID="1fc4788dddef4992b313771aff7147eb44bbea4601280a067bc1b3feb7ec010a" exitCode=0
Oct 30 18:39:21.523203 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.522914    6014 generic.go:356] "Generic (PLEG): container finished" podID="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" containerID="c7c7bba50f1532576ee01b575f7356ffa55201874d30d02c3259210f96a27206" exitCode=0
Oct 30 18:39:21.523203 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.522922    6014 generic.go:356] "Generic (PLEG): container finished" podID="b2e259e9-ce5d-458d-8b2f-7a5e4448e587" containerID="10c6d83d51c796d56975d832d24b7e46c8e17f4228223d295c501ad48a887746" exitCode=0
Oct 30 18:39:21.533393 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.533327    6014 generic.go:356] "Generic (PLEG): container finished" podID="5d438d4b-2388-4a65-a199-8cb4b3d27dc2" containerID="3ef2e2ef41957189b82a5df17679842dc281332bd5b5eb6948b9abd106615c9b" exitCode=0
Oct 30 18:39:21.538399 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:21.538372    6014 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-97.us-west-1.compute.internal\" not found"
Oct 30 18:39:21.541472 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.541447    6014 generic.go:356] "Generic (PLEG): container finished" podID="349d28f3fc367c9ffa500d3d4a12aae5" containerID="5ec01d92ca7eb5ff7183b4a6d17b942ca0d98ca6333f662dbdac96f4efe207c2" exitCode=0
Oct 30 18:39:21.568261 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:21.568224    6014 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 18:39:21.570358 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.570335    6014 manager.go:324] Recovery completed
Oct 30 18:39:21.594485 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.594460    6014 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:39:21.597547 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.597528    6014 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:39:21.597653 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.597563    6014 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:39:21.597653 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.597576    6014 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:39:21.600731 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.600710    6014 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:39:21.600731 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.600724    6014 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:39:21.600896 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.600740    6014 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:39:21.600896 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.600748    6014 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:39:21.600997 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.600911    6014 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 18:39:21.600997 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.600926    6014 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 18:39:21.600997 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.600951    6014 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 18:39:21.600997 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.600960    6014 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 18:39:21.600997 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.600968    6014 policy_none.go:50] "Start"
Oct 30 18:39:21.600997 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.600981    6014 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:39:21.600997 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.600993    6014 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:39:21.601328 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.601201    6014 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 18:39:21.601328 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.601213    6014 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 18:39:21.601328 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.601222    6014 policy_none.go:44] "Start"
Oct 30 18:39:21.601851 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.601834    6014 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:39:21.601933 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.601868    6014 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:39:21.601933 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.601884    6014 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:39:21.606398 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.606373    6014 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:39:21.608939 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.608923    6014 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:39:21.609005 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:21.608947    6014 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:39:21.609005 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.608956    6014 server.go:87] "Starting device plugin registration server"
Oct 30 18:39:21.609303 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.609290    6014 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:39:21.609335 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.609306    6014 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:39:21.609385 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.609369    6014 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:39:21.609508 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.609463    6014 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:39:21.609627 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.609512    6014 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:39:21.609627 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.609519    6014 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:39:21.610160 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.609552    6014 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:39:21.609502043Z","UUID":"7c59d1ac-38ff-4429-b5fd-3484395ecffd","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:39:21.615512 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:21.615483    6014 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:39:21.615615 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:21.615537    6014 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-97.us-west-1.compute.internal\" not found"
Oct 30 18:39:21.615928 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.615910    6014 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:39:21.615982 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.615945    6014 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:39:21.628399 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.628379    6014 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:39:21.628518 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.628409    6014 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:39:21.651942 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.651916    6014 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 18:39:21.651942 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.651935    6014 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:39:21.709560 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.709533    6014 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:39:21.710717 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.710696    6014 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:39:21.710822 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.710730    6014 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:39:21.710822 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.710740    6014 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-97.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:39:21.710822 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.710792    6014 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.726347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.726328    6014 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:39:21.768394 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.768348    6014 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"]
Oct 30 18:39:21.769074 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.769053    6014 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.769206 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.769088    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal" event={"ID":"5a238862e2cf33c9dc387d53f60c786c","Type":"ContainerStarted","Data":"73ed0227f00f949e0a46e74ccfa42589f3069c5c9a600863dbde210f1d9b02f8"}
Oct 30 18:39:21.769206 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.769124    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal" event={"ID":"5a238862e2cf33c9dc387d53f60c786c","Type":"ContainerStarted","Data":"8f0941b9270fcdcf0be5202685ed7f97e1a19a72fb544148ab8ce523fad9437f"}
Oct 30 18:39:21.769206 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.769135    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal" event={"ID":"349d28f3fc367c9ffa500d3d4a12aae5","Type":"ContainerStarted","Data":"7da45b4d8dd8a7950fce51a3bf0d66cc0b6ec01a425049e8fb55ca2915f34a57"}
Oct 30 18:39:21.769206 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.769172    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal" event={"ID":"349d28f3fc367c9ffa500d3d4a12aae5","Type":"ContainerDied","Data":"5ec01d92ca7eb5ff7183b4a6d17b942ca0d98ca6333f662dbdac96f4efe207c2"}
Oct 30 18:39:21.769206 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.769184    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal" event={"ID":"349d28f3fc367c9ffa500d3d4a12aae5","Type":"ContainerStarted","Data":"3309f32908643a7ebc30a95f6d73c02dfdfe2734a33d056980f0e964cdf091b4"}
Oct 30 18:39:21.769387 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.769283    6014 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.819488 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.819463    6014 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.819638 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.819531    6014 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.830792 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.830774    6014 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:39:21.830918 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:21.830821    6014 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.831128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.831114    6014 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:39:21.831206 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:21.831170    6014 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.853194 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.853169    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/349d28f3fc367c9ffa500d3d4a12aae5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"349d28f3fc367c9ffa500d3d4a12aae5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.853313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.853212    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/349d28f3fc367c9ffa500d3d4a12aae5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"349d28f3fc367c9ffa500d3d4a12aae5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.853313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.853238    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a238862e2cf33c9dc387d53f60c786c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"5a238862e2cf33c9dc387d53f60c786c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.954361 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.954328    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/349d28f3fc367c9ffa500d3d4a12aae5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"349d28f3fc367c9ffa500d3d4a12aae5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.954521 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.954369    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/349d28f3fc367c9ffa500d3d4a12aae5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"349d28f3fc367c9ffa500d3d4a12aae5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.954521 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.954395    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a238862e2cf33c9dc387d53f60c786c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"5a238862e2cf33c9dc387d53f60c786c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.954521 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.954439    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a238862e2cf33c9dc387d53f60c786c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"5a238862e2cf33c9dc387d53f60c786c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.954521 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.954439    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/349d28f3fc367c9ffa500d3d4a12aae5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"349d28f3fc367c9ffa500d3d4a12aae5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:21.954521 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:21.954439    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/349d28f3fc367c9ffa500d3d4a12aae5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" (UID: \"349d28f3fc367c9ffa500d3d4a12aae5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:22.422628 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.422583    6014 apiserver.go:52] "Watching apiserver"
Oct 30 18:39:22.525710 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.525680    6014 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:39:22.526455 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.526427    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-plh42","kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal","openshift-console/console-6bbbd6864b-w27vc","openshift-network-diagnostics/network-check-target-27m75","openshift-network-operator/iptables-alerter-ctrbm","openshift-dns/node-resolver-7mfxz","openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc","openshift-monitoring/telemeter-client-78cd67666d-td5fr","openshift-multus/network-metrics-daemon-676kr","openshift-monitoring/alertmanager-main-0","openshift-ingress-canary/ingress-canary-7rxxp","kube-system/global-pull-secret-syncer-7h2cm","openshift-dns/dns-default-4kspp","openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9","kube-system/konnectivity-agent-wwvr8","openshift-monitoring/metrics-server-7848bc9d85-wmsb2","openshift-monitoring/node-exporter-h2x92","openshift-multus/multus-pkq65","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-additional-cni-plugins-6bjz2","openshift-ovn-kubernetes/ovnkube-node-54qsj","openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7","openshift-image-registry/node-ca-8pwf7","openshift-insights/insights-runtime-extractor-fzvff","openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"]
Oct 30 18:39:22.545150 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.545112    6014 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:22.545286 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.545270    6014 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:22.558510 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558485    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1395368e-c046-4067-809a-6310f8d3c93d-host-slash\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:39:22.558595 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558516    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:39:22.558595 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558540    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf7720c-65a8-494e-a679-239f3a59ba3f-tmp-dir\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:39:22.558595 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558555    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqnx\" (UniqueName: \"kubernetes.io/projected/bdf7720c-65a8-494e-a679-239f3a59ba3f-kube-api-access-5hqnx\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:39:22.558706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558618    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-system-cni-dir\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.558706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558676    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-systemd\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.558764 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558726    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf7720c-65a8-494e-a679-239f3a59ba3f-tmp-dir\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:39:22.558793 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558759    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-cni-bin\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.558825 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558791    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1395368e-c046-4067-809a-6310f8d3c93d-iptables-alerter-script\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:39:22.558825 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558810    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crdr9\" (UniqueName: \"kubernetes.io/projected/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-kube-api-access-crdr9\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:39:22.558881 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558838    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-run-netns\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.558913 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558902    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/717a2238-075b-4288-9c74-1a306efb331d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:39:22.558946 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558929    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/45c93bbd-c046-4a50-83f2-ae8de8b64104-hosts-file\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:39:22.558975 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558947    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-run-ovn-kubernetes\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.558975 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558962    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-netns\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.559031 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.558990    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-device-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.559059 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559026    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-kubelet-config\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:39:22.559092 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559066    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbtd6\" (UniqueName: \"kubernetes.io/projected/1395368e-c046-4067-809a-6310f8d3c93d-kube-api-access-fbtd6\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:39:22.559121 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559094    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-ovnkube-script-lib\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.559196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559181    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-host\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.559228 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559212    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-hostroot\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.559263 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559251    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-multus-certs\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.559299 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559275    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/717a2238-075b-4288-9c74-1a306efb331d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:39:22.559329 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559307    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj64r\" (UniqueName: \"kubernetes.io/projected/45c93bbd-c046-4a50-83f2-ae8de8b64104-kube-api-access-gj64r\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:39:22.559369 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559345    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-data-volume\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:39:22.559397 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559369    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:39:22.559476 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559461    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-slash\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.559510 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559484    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysctl-d\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.559510 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559498    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-data-volume\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:39:22.559563 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559543    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-textfile\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.559595 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559563    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.559624 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559603    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-textfile\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.559655 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559630    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.559707 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559690    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysconfig\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.559748 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559736    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-socket-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.559782 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559758    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvfqq\" (UniqueName: \"kubernetes.io/projected/0b629f13-9f3f-41a9-9122-5d9fbe050beb-kube-api-access-vvfqq\") pod \"ingress-canary-7rxxp\" (UID: \"0b629f13-9f3f-41a9-9122-5d9fbe050beb\") " pod="openshift-ingress-canary/ingress-canary-7rxxp"
Oct 30 18:39:22.559853 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559805    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.559883 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559875    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-system-cni-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.559944 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559928    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-cni-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.559976 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559966    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.560009 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.559989    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xs2k\" (UniqueName: \"kubernetes.io/projected/a6cfb91c-11df-4d59-b723-320754d81b99-kube-api-access-5xs2k\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.560041 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560025    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-node-log\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.560123 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560112    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-cni-multus\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.560177 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560150    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkl46\" (UniqueName: \"kubernetes.io/projected/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-api-access-gkl46\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:39:22.560177 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560166    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/594ac1de-c6f2-4031-807e-08509c8a162d-serviceca\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:39:22.560245 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560223    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-systemd-units\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.560278 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560263    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-cni-netd\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.560319 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560306    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-cnibin\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.560349 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560337    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-etc-selinux\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.560378 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560359    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.560415 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560398    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-accelerators-collector-config\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.560472 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560458    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhfj5\" (UniqueName: \"kubernetes.io/projected/717a2238-075b-4288-9c74-1a306efb331d-kube-api-access-xhfj5\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:39:22.560588 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560576    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.560646 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560632    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.560680 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560655    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-sys\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.560680 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560672    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a1ed51f8-138a-4e90-bcfe-00cae5eba235-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:39:22.560741 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560690    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf5zd\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-kube-api-access-bf5zd\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.560769 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560736    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6cfb91c-11df-4d59-b723-320754d81b99-tmp\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.560769 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560750    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a1ed51f8-138a-4e90-bcfe-00cae5eba235-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:39:22.560769 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560757    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-registration-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.560852 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560783    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf7720c-65a8-494e-a679-239f3a59ba3f-config-volume\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:39:22.560852 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560809    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-run\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.560852 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560827    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-sys-fs\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.560852 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560847    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.560962 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560868    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6cfb91c-11df-4d59-b723-320754d81b99-tmp\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.560962 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560876    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-log-socket\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.560962 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560932    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-cni-bin\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.561070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560960    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db2ck\" (UniqueName: \"kubernetes.io/projected/594ac1de-c6f2-4031-807e-08509c8a162d-kube-api-access-db2ck\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:39:22.561070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560977    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d70a762-bdea-4394-8a95-5e5f0fb4042c-agent-certs\") pod \"konnectivity-agent-wwvr8\" (UID: \"4d70a762-bdea-4394-8a95-5e5f0fb4042c\") " pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:39:22.561070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.560992    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.561070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561016    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-wtmp\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.561070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561036    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/594ac1de-c6f2-4031-807e-08509c8a162d-host\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:39:22.561070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561052    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:39:22.561070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561068    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.561372 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561095    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:39:22.561372 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561159    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.561372 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561188    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.561372 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561206    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f478498d-4adf-441d-9dc6-1988d7c9e050-ovn-node-metrics-cert\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.561372 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561224    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-kubelet\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.561372 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561245    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-web-config\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.561372 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561301    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.561372 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561317    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-etc-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.561372 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561344    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-ovnkube-config\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.561706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561378    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f95bb75f-1760-4132-bf36-20c2027b3059-cni-binary-copy\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.561706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561421    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-k8s-cni-cncf-io\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.561706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561448    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6vzs\" (UniqueName: \"kubernetes.io/projected/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-kube-api-access-b6vzs\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.561706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561466    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8lg\" (UniqueName: \"kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg\") pod \"network-check-target-27m75\" (UID: \"761ddebb-9140-4814-8145-3bb12ce8a148\") " pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:39:22.561706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561486    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-out\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.561706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561535    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-out\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.561706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561586    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-conf-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.561706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561607    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgcm\" (UniqueName: \"kubernetes.io/projected/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-kube-api-access-7vgcm\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.561706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561633    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.561706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561684    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.562070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561702    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.562070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561731    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-kubelet\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.562070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561746    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-var-lib-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.562070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561762    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-kubernetes\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.562070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561869    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d70a762-bdea-4394-8a95-5e5f0fb4042c-konnectivity-ca\") pod \"konnectivity-agent-wwvr8\" (UID: \"4d70a762-bdea-4394-8a95-5e5f0fb4042c\") " pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:39:22.562070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561898    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f95bb75f-1760-4132-bf36-20c2027b3059-multus-daemon-config\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.562070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561915    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-root\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.562070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561960    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6cfb91c-11df-4d59-b723-320754d81b99-metrics-client-ca\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.562070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.561986    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cnibin\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.562070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562002    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.562070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562017    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-modprobe-d\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.562070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562056    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/028108f5-f006-42d0-9cfd-56f9b55e18ff-tmp\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562091    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjnqm\" (UniqueName: \"kubernetes.io/projected/f95bb75f-1760-4132-bf36-20c2027b3059-kube-api-access-qjnqm\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562117    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-tls\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562128    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/028108f5-f006-42d0-9cfd-56f9b55e18ff-tmp\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562169    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562192    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562206    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-env-overrides\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562224    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562286    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562312    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzx9l\" (UniqueName: \"kubernetes.io/projected/f478498d-4adf-441d-9dc6-1988d7c9e050-kube-api-access-jzx9l\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562328    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-tuned\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562376    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-tuned\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562426    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-etc-kubernetes\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562448    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:39:22.562477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562478    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:39:22.562847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562500    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhvz6\" (UniqueName: \"kubernetes.io/projected/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-kube-api-access-qhvz6\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:39:22.562847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562529    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862sw\" (UniqueName: \"kubernetes.io/projected/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-kube-api-access-862sw\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.562847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562555    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysctl-conf\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.562847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562585    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2r6v\" (UniqueName: \"kubernetes.io/projected/028108f5-f006-42d0-9cfd-56f9b55e18ff-kube-api-access-d2r6v\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.562847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562609    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-dbus\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:39:22.562847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562646    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-crio-socket\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:39:22.562847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562713    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.562847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562743    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-var-lib-kubelet\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.562847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562760    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.562847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562783    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed51f8-138a-4e90-bcfe-00cae5eba235-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:39:22.562847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562806    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c93bbd-c046-4a50-83f2-ae8de8b64104-tmp-dir\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:39:22.562847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562839    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-grpc-tls\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.563172 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562953    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c93bbd-c046-4a50-83f2-ae8de8b64104-tmp-dir\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:39:22.563172 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562955    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-lib-modules\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.563172 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.562994    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-socket-dir-parent\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.563172 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.563028    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdf7720c-65a8-494e-a679-239f3a59ba3f-metrics-tls\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:39:22.563172 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.563100    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-sys\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.563172 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.563132    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-tls\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.563355 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.563204    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-systemd\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.563355 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.563228    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-ovn\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.563355 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.563246    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-metrics-client-ca\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.563355 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.563264    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b629f13-9f3f-41a9-9122-5d9fbe050beb-cert\") pod \"ingress-canary-7rxxp\" (UID: \"0b629f13-9f3f-41a9-9122-5d9fbe050beb\") " pod="openshift-ingress-canary/ingress-canary-7rxxp"
Oct 30 18:39:22.563355 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.563294    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.563506 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.563371    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/717a2238-075b-4288-9c74-1a306efb331d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:39:22.580017 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.579988    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:39:22.580017 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.580019    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:39:22.580253 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.580033    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:39:22.580253 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.580056    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:39:22.580253 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.580175    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:39:22.580253 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.580194    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:39:22.580253 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.580241    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:39:22.580253 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.580243    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:39:22.580524 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.580279    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:39:22.580524 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.580283    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:39:22.580524 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.580409    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:39:22.580524 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.580419    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:39:22.580634 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.580529    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:39:22.580634 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.580611    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:39:22.581463 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.581446    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.581603 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.581591    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f95bb75f-1760-4132-bf36-20c2027b3059-cni-binary-copy\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.582855 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.582840    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f95bb75f-1760-4132-bf36-20c2027b3059-multus-daemon-config\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.583163 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.583128    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:39:22.583281 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.583271    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:39:22.583786 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.583775    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:39:22.584087 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.584072    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:39:22.584169 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.584094    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:39:22.584530 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.584515    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:39:22.584601 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.584590    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:39:22.584738 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.584728    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:39:22.590072 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.590056    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1395368e-c046-4067-809a-6310f8d3c93d-iptables-alerter-script\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:39:22.591456 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.591436    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/594ac1de-c6f2-4031-807e-08509c8a162d-serviceca\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:39:22.591619 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.591604    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-ovnkube-config\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.591619 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.591611    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.592290 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.592275    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d70a762-bdea-4394-8a95-5e5f0fb4042c-konnectivity-ca\") pod \"konnectivity-agent-wwvr8\" (UID: \"4d70a762-bdea-4394-8a95-5e5f0fb4042c\") " pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:39:22.593438 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.593424    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.618685 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.618661    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:39:22.631285 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.631260    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbtd6\" (UniqueName: \"kubernetes.io/projected/1395368e-c046-4067-809a-6310f8d3c93d-kube-api-access-fbtd6\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:39:22.642875 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.642172    6014 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:39:22.643485 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.643460    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 18:39:22.643614 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.643597    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 18:39:22.643686 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.643616    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 18:39:22.643686 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.643638    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 18:39:22.643686 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.643621    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 18:39:22.643819 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.643718    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 18:39:22.643819 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.643734    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 18:39:22.643819 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.643742    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 18:39:22.643819 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.643774    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 18:39:22.643956 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.643840    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 18:39:22.643956 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.643875    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:39:22.643956 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.643876    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:39:22.643956 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.643890    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kbv4s\""
Oct 30 18:39:22.644132 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644037    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 18:39:22.644132 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644045    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 18:39:22.644132 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644118    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 18:39:22.644274 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644195    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:39:22.644274 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644203    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 18:39:22.644375 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644280    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bftj9\""
Oct 30 18:39:22.644375 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644341    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:39:22.644375 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644368    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:39:22.644507 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644377    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 18:39:22.644507 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644385    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 18:39:22.644507 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644412    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 18:39:22.644507 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644414    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 18:39:22.644507 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644453    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 18:39:22.644507 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644460    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 18:39:22.644507 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644461    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 18:39:22.644507 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644487    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 18:39:22.644507 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644491    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 18:39:22.644507 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644491    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 18:39:22.644843 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644537    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xhgds\""
Oct 30 18:39:22.644843 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644714    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-476j6\""
Oct 30 18:39:22.644843 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644749    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xqkxc\""
Oct 30 18:39:22.644843 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644820    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bhm78\""
Oct 30 18:39:22.644843 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644825    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 18:39:22.644843 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644841    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644849    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q8mbb\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644822    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xlrq8\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644882    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-cts7j\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644890    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644918    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644938    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644946    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gk56f\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644940    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644975    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.644985    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m9qwf\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645002    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645021    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-w27xd\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645028    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-m8bck\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645063    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bh5pq\""
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645068    6014 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:39:22.645112 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.645108    6014 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645127    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645068    6014 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645243    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2obehff8v19io\""
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645276    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xnv76\""
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645307    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8668tapl4mk7f\""
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645344    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lx4z8\""
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645423    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nsz9w\""
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645463    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db2ck\" (UniqueName: \"kubernetes.io/projected/594ac1de-c6f2-4031-807e-08509c8a162d-kube-api-access-db2ck\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645491    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2r6v\" (UniqueName: \"kubernetes.io/projected/028108f5-f006-42d0-9cfd-56f9b55e18ff-kube-api-access-d2r6v\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645496    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645560    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6nrlk\""
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645617    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bdjh3qnjck7pd\""
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645698    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9gxzb\""
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645720    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s4fk8\""
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645755    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zdt4f\""
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645797    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h468w\""
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645818    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crdr9\" (UniqueName: \"kubernetes.io/projected/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-kube-api-access-crdr9\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:39:22.645877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645880    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-862sw\" (UniqueName: \"kubernetes.io/projected/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-kube-api-access-862sw\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.646539 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645904    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgcm\" (UniqueName: \"kubernetes.io/projected/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-kube-api-access-7vgcm\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.646539 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645929    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 18:39:22.646539 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.645966    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8wzk6\""
Oct 30 18:39:22.646539 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.646037    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 18:39:22.646539 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.646070    6014 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal"
Oct 30 18:39:22.646539 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.646103    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 18:39:22.646539 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.646243    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:39:22.646539 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.646261    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjnqm\" (UniqueName: \"kubernetes.io/projected/f95bb75f-1760-4132-bf36-20c2027b3059-kube-api-access-qjnqm\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.646539 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.646324    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dx4x5\""
Oct 30 18:39:22.646539 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.646450    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzx9l\" (UniqueName: \"kubernetes.io/projected/f478498d-4adf-441d-9dc6-1988d7c9e050-kube-api-access-jzx9l\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.646802 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.646572    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r8lg\" (UniqueName: \"kubernetes.io/projected/761ddebb-9140-4814-8145-3bb12ce8a148-kube-api-access-4r8lg\") pod \"network-check-target-27m75\" (UID: \"761ddebb-9140-4814-8145-3bb12ce8a148\") " pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:39:22.649545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.649530    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/717a2238-075b-4288-9c74-1a306efb331d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:39:22.649641 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.649610    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-original-pull-secret\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:39:22.649686 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.649660    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:39:22.650825 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.650807    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.650900 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.650884    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.650934 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.650899    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.651005 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.650947    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.651057 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.651043    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.651225 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.651210    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d70a762-bdea-4394-8a95-5e5f0fb4042c-agent-certs\") pod \"konnectivity-agent-wwvr8\" (UID: \"4d70a762-bdea-4394-8a95-5e5f0fb4042c\") " pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:39:22.651364 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.651351    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.651397 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.651355    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.651397 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.651371    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8659e3f-b750-4d38-aa0b-7cc7f883f9b4-metrics-certs\") pod \"network-metrics-daemon-676kr\" (UID: \"e8659e3f-b750-4d38-aa0b-7cc7f883f9b4\") " pod="openshift-multus/network-metrics-daemon-676kr"
Oct 30 18:39:22.651465 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.651452    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-web-config\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.651494 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.651484    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f478498d-4adf-441d-9dc6-1988d7c9e050-ovn-node-metrics-cert\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.652038 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.652020    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.652105 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.652073    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.652209 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.652194    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.653310 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.653292    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:39:22.653378 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.653319    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:39:22.653378 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.653335    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-grpc-tls\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.653378 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.653346    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdf7720c-65a8-494e-a679-239f3a59ba3f-metrics-tls\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:39:22.653475 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.653391    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.653475 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.653421    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-tls\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.653475 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.653448    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a6cfb91c-11df-4d59-b723-320754d81b99-secret-thanos-querier-tls\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.653475 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.653460    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.653589 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.653534    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/717a2238-075b-4288-9c74-1a306efb331d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:39:22.653589 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.653536    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b629f13-9f3f-41a9-9122-5d9fbe050beb-cert\") pod \"ingress-canary-7rxxp\" (UID: \"0b629f13-9f3f-41a9-9122-5d9fbe050beb\") " pod="openshift-ingress-canary/ingress-canary-7rxxp"
Oct 30 18:39:22.664325 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664309    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-socket-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.664381 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664333    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.664381 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664350    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-system-cni-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.664381 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664365    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-cni-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.664479 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664380    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:22.664479 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664402    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpn9\" (UniqueName: \"kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:22.664479 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664431    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-system-cni-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.664479 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664438    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.664479 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664431    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-socket-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.664479 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664448    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-cni-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.664479 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664467    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.664479 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.664467    6014 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 18:39:22.664747 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664565    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-node-log\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.664747 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664581    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-cni-multus\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.664747 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664607    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-federate-client-tls\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.664747 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664627    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-node-log\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.664747 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664633    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vd7v\" (UniqueName: \"kubernetes.io/projected/6cebbc4c-d198-4222-968d-4ab6d035ee51-kube-api-access-9vd7v\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.664747 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664637    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-cni-multus\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.664747 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664660    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.664747 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.664698    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:23.164652877 +0000 UTC m=+1.807820477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-config" not registered
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664748    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664789    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-systemd-units\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664810    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-federate-client-tls\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664817    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-cni-netd\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664841    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-cnibin\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664845    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-systemd-units\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664865    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-etc-selinux\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664846    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-cni-netd\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664878    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-cnibin\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664894    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664922    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-sys\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664948    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664955    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-etc-selinux\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664966    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664978    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-registration-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.664992    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-sys\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665006    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665031    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665058    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-client-ca-bundle\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665081    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-run\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665082    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-registration-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.665090    6014 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665107    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-sys-fs\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665183    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-run\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.665175    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:23.165158018 +0000 UTC m=+1.808325630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 18:39:22.665196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665206    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-sys-fs\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665227    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665226    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-log-socket\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665271    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-cni-bin\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665275    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-log-socket\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665279    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-client-ca-bundle\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665300    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665318    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665335    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-wtmp\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665334    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-cni-bin\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665350    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/594ac1de-c6f2-4031-807e-08509c8a162d-host\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665367    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73dec725-685b-4a6d-81b4-783f6c79494e-tmp\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665390    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909bd6eb-4140-4939-a5c1-4d148b3791dc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gbbdc\" (UID: \"909bd6eb-4140-4939-a5c1-4d148b3791dc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665408    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.665412    6014 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665427    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-kubelet\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665393    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/594ac1de-c6f2-4031-807e-08509c8a162d-host\") pod \"node-ca-8pwf7\" (UID: \"594ac1de-c6f2-4031-807e-08509c8a162d\") " pod="openshift-image-registry/node-ca-8pwf7"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665451    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665471    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-wtmp\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665477    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665487    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-kubelet\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665520    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-etc-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665440    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73dec725-685b-4a6d-81b4-783f6c79494e-tmp\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665515    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.665533    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:23.165518602 +0000 UTC m=+1.808686210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665559    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909bd6eb-4140-4939-a5c1-4d148b3791dc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gbbdc\" (UID: \"909bd6eb-4140-4939-a5c1-4d148b3791dc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665569    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-etc-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665614    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-k8s-cni-cncf-io\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665627    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665645    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665663    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-k8s-cni-cncf-io\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665667    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665674    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665700    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665725    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665757    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-conf-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665760    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.665763    6014 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665779    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665787    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665810    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-conf-dir\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665817    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-kubelet\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.665826    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:23.165813817 +0000 UTC m=+1.808981432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665859    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-kubelet\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665861    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-var-lib-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665892    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-kubernetes\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665903    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665921    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665922    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-var-lib-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665943    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665981    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-root\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665986    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.665938    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-kubernetes\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666007    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-root\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666017    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-config\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666037    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cnibin\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666044    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666052    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666071    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-modprobe-d\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666079    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-openvswitch\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666092    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666114    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666107    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-cnibin\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.666134    6014 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666205    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-modprobe-d\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666133    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.666236    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:23.166219279 +0000 UTC m=+1.809386871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"service-ca" not registered
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666264    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-config\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666280    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-etc-kubernetes\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.666280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666312    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-secret-metrics-server-client-certs\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666332    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-etc-kubernetes\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666336    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666340    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666364    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysctl-conf\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666381    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-dbus\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666397    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6cebbc4c-d198-4222-968d-4ab6d035ee51-audit-log\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666413    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-crio-socket\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666428    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-var-lib-kubelet\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666469    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666479    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-var-lib-kubelet\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666484    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-crio-socket\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666489    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666471    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-dbus\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666493    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysctl-conf\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666502    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-serving-certs-ca-bundle\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666512    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-secret-metrics-server-client-certs\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666534    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6cebbc4c-d198-4222-968d-4ab6d035ee51-audit-log\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666542    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666545    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666570    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-lib-modules\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666585    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-socket-dir-parent\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666601    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-sys\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.666618    6014 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666654    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-sys\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666665    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-lib-modules\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.666659    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:23.166645693 +0000 UTC m=+1.809813306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666618    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666700    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6cebbc4c-d198-4222-968d-4ab6d035ee51-metrics-server-audit-profiles\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666719    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-systemd\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666652    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-multus-socket-dir-parent\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666733    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-ovn\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666743    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666755    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-telemeter-client-tls\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666764    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-systemd\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666774    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1395368e-c046-4067-809a-6310f8d3c93d-host-slash\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666781    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-run-ovn\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666797    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1395368e-c046-4067-809a-6310f8d3c93d-host-slash\") pod \"iptables-alerter-ctrbm\" (UID: \"1395368e-c046-4067-809a-6310f8d3c93d\") " pod="openshift-network-operator/iptables-alerter-ctrbm"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666806    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjkqx\" (UniqueName: \"kubernetes.io/projected/2cf0ddaf-aa3c-4390-b911-cab27953af11-kube-api-access-cjkqx\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666834    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-system-cni-dir\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666858    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-systemd\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666883    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-cni-bin\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666906    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-var-lib-cni-bin\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666910    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-run-netns\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666919    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-systemd\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666921    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-telemeter-client-tls\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666932    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-run-netns\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666885    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e259e9-ce5d-458d-8b2f-7a5e4448e587-system-cni-dir\") pod \"multus-additional-cni-plugins-6bjz2\" (UID: \"b2e259e9-ce5d-458d-8b2f-7a5e4448e587\") " pod="openshift-multus/multus-additional-cni-plugins-6bjz2"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666940    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-secret-metrics-server-tls\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.666984    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/45c93bbd-c046-4a50-83f2-ae8de8b64104-hosts-file\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667016    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-run-ovn-kubernetes\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667040    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-netns\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667072    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/45c93bbd-c046-4a50-83f2-ae8de8b64104-hosts-file\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667084    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-run-ovn-kubernetes\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667086    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-device-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667113    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-kubelet-config\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667121    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-netns\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667129    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cebbc4c-d198-4222-968d-4ab6d035ee51-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667159    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae5a6b4e-719e-4978-afa3-8d179bbcbdd2-device-dir\") pod \"aws-ebs-csi-driver-node-74mtn\" (UID: \"ae5a6b4e-719e-4978-afa3-8d179bbcbdd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74mtn"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667173    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667193    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-host\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667193    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab2168bb-3bf1-48ed-993d-4d0a014f00ae-kubelet-config\") pod \"global-pull-secret-syncer-7h2cm\" (UID: \"ab2168bb-3bf1-48ed-993d-4d0a014f00ae\") " pod="kube-system/global-pull-secret-syncer-7h2cm"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667209    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-hostroot\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667225    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-multus-certs\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667227    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6cebbc4c-d198-4222-968d-4ab6d035ee51-secret-metrics-server-tls\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667236    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-host\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667254    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-host-run-multus-certs\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667260    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f95bb75f-1760-4132-bf36-20c2027b3059-hostroot\") pod \"multus-pkq65\" (UID: \"f95bb75f-1760-4132-bf36-20c2027b3059\") " pod="openshift-multus/multus-pkq65"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667272    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-secret-telemeter-client\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667290    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-metrics-client-ca\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667309    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-slash\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667331    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f478498d-4adf-441d-9dc6-1988d7c9e050-host-slash\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667357    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysctl-d\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667375    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg7ts\" (UniqueName: \"kubernetes.io/projected/909bd6eb-4140-4939-a5c1-4d148b3791dc-kube-api-access-qg7ts\") pod \"monitoring-plugin-77f4f76594-gbbdc\" (UID: \"909bd6eb-4140-4939-a5c1-4d148b3791dc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667391    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtzzk\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-kube-api-access-qtzzk\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667407    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysconfig\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667453    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysconfig\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667477    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2cf0ddaf-aa3c-4390-b911-cab27953af11-secret-telemeter-client\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.668826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.667477    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/028108f5-f006-42d0-9cfd-56f9b55e18ff-etc-sysctl-d\") pod \"tuned-plh42\" (UID: \"028108f5-f006-42d0-9cfd-56f9b55e18ff\") " pod="openshift-cluster-node-tuning-operator/tuned-plh42"
Oct 30 18:39:22.679967 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.679916    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:39:22.680091 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680076    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:39:22.680153 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680101    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:39:22.680193 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680169    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:39:22.680242 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680223    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:39:22.680288 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680266    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 18:39:22.680288 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680266    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:39:22.680381 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680287    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 18:39:22.680381 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680373    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 18:39:22.680463 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680391    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 18:39:22.680463 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680403    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 18:39:22.680528 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680505    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 18:39:22.680677 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680666    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:39:22.680723 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680711    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:39:22.680769 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680746    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 18:39:22.680769 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680761    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:39:22.680859 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680803    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:39:22.680859 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.680831    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:39:22.681078 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.681056    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf7720c-65a8-494e-a679-239f3a59ba3f-config-volume\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:39:22.681337 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.681297    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:39:22.681486 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.681463    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj64r\" (UniqueName: \"kubernetes.io/projected/45c93bbd-c046-4a50-83f2-ae8de8b64104-kube-api-access-gj64r\") pod \"node-resolver-7mfxz\" (UID: \"45c93bbd-c046-4a50-83f2-ae8de8b64104\") " pod="openshift-dns/node-resolver-7mfxz"
Oct 30 18:39:22.681609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.681573    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkl46\" (UniqueName: \"kubernetes.io/projected/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-api-access-gkl46\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:39:22.681660 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.681610    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqnx\" (UniqueName: \"kubernetes.io/projected/bdf7720c-65a8-494e-a679-239f3a59ba3f-kube-api-access-5hqnx\") pod \"dns-default-4kspp\" (UID: \"bdf7720c-65a8-494e-a679-239f3a59ba3f\") " pod="openshift-dns/dns-default-4kspp"
Oct 30 18:39:22.681660 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.681610    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xs2k\" (UniqueName: \"kubernetes.io/projected/a6cfb91c-11df-4d59-b723-320754d81b99-kube-api-access-5xs2k\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.682438 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.682423    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-env-overrides\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.683529 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.683510    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ed51f8-138a-4e90-bcfe-00cae5eba235-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:39:22.684584 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.684570    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.685707 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.685687    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.685861 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.685841    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.686026 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.686011    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6vzs\" (UniqueName: \"kubernetes.io/projected/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-kube-api-access-b6vzs\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.686127 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.686112    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvfqq\" (UniqueName: \"kubernetes.io/projected/0b629f13-9f3f-41a9-9122-5d9fbe050beb-kube-api-access-vvfqq\") pod \"ingress-canary-7rxxp\" (UID: \"0b629f13-9f3f-41a9-9122-5d9fbe050beb\") " pod="openshift-ingress-canary/ingress-canary-7rxxp"
Oct 30 18:39:22.686204 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.686126    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhvz6\" (UniqueName: \"kubernetes.io/projected/ac09d41c-10f4-4d5d-a8cb-1e638a70ab35-kube-api-access-qhvz6\") pod \"insights-runtime-extractor-fzvff\" (UID: \"ac09d41c-10f4-4d5d-a8cb-1e638a70ab35\") " pod="openshift-insights/insights-runtime-extractor-fzvff"
Oct 30 18:39:22.686204 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.686177    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf5zd\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-kube-api-access-bf5zd\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.686324 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.686308    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.686697 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.686680    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhfj5\" (UniqueName: \"kubernetes.io/projected/717a2238-075b-4288-9c74-1a306efb331d-kube-api-access-xhfj5\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:39:22.686848 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.686833    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-serving-certs-ca-bundle\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.686954 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.686942    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6cebbc4c-d198-4222-968d-4ab6d035ee51-metrics-server-audit-profiles\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.688052 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.688035    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cebbc4c-d198-4222-968d-4ab6d035ee51-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.691298 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.691283    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-node-exporter-accelerators-collector-config\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.693319 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.693264    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:39:22.693555 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.693537    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d438d4b-2388-4a65-a199-8cb4b3d27dc2-metrics-client-ca\") pod \"node-exporter-h2x92\" (UID: \"5d438d4b-2388-4a65-a199-8cb4b3d27dc2\") " pod="openshift-monitoring/node-exporter-h2x92"
Oct 30 18:39:22.693718 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.693706    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:39:22.697774 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.697625    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 18:39:22.697774 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.697742    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-metrics-client-ca\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.697774 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.697758    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 18:39:22.697774 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.697773    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.698205 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.698187    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 18:39:22.698247 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.698187    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 18:39:22.699477 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.699458    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/717a2238-075b-4288-9c74-1a306efb331d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jbhtt\" (UID: \"717a2238-075b-4288-9c74-1a306efb331d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jbhtt"
Oct 30 18:39:22.699532 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.699488    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f478498d-4adf-441d-9dc6-1988d7c9e050-ovnkube-script-lib\") pod \"ovnkube-node-54qsj\" (UID: \"f478498d-4adf-441d-9dc6-1988d7c9e050\") " pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:22.700624 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.700608    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.700762 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.700698    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:22.702842 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.702822    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6cfb91c-11df-4d59-b723-320754d81b99-metrics-client-ca\") pod \"thanos-querier-68b7f4fd8f-mttl7\" (UID: \"a6cfb91c-11df-4d59-b723-320754d81b99\") " pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:22.703974 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.703955    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed51f8-138a-4e90-bcfe-00cae5eba235-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2fxg9\" (UID: \"a1ed51f8-138a-4e90-bcfe-00cae5eba235\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fxg9"
Oct 30 18:39:22.705844 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.705824    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.709762 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.706772    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf0ddaf-aa3c-4390-b911-cab27953af11-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.710332 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.710314    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:22.719205 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.719189    6014 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 18:39:22.719257 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.719207    6014 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 18:39:22.719257 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.719219    6014 projected.go:196] Error preparing data for projected volume kube-api-access-7lpn9 for pod openshift-console/console-6bbbd6864b-w27vc: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 18:39:22.719324 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:22.719281    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9 podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:23.219267694 +0000 UTC m=+1.862435287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7lpn9" (UniqueName: "kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 18:39:22.719545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.719528    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vd7v\" (UniqueName: \"kubernetes.io/projected/6cebbc4c-d198-4222-968d-4ab6d035ee51-kube-api-access-9vd7v\") pod \"metrics-server-7848bc9d85-wmsb2\" (UID: \"6cebbc4c-d198-4222-968d-4ab6d035ee51\") " pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:22.720578 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.720561    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjkqx\" (UniqueName: \"kubernetes.io/projected/2cf0ddaf-aa3c-4390-b911-cab27953af11-kube-api-access-cjkqx\") pod \"telemeter-client-78cd67666d-td5fr\" (UID: \"2cf0ddaf-aa3c-4390-b911-cab27953af11\") " pod="openshift-monitoring/telemeter-client-78cd67666d-td5fr"
Oct 30 18:39:22.720972 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.720956    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg7ts\" (UniqueName: \"kubernetes.io/projected/909bd6eb-4140-4939-a5c1-4d148b3791dc-kube-api-access-qg7ts\") pod \"monitoring-plugin-77f4f76594-gbbdc\" (UID: \"909bd6eb-4140-4939-a5c1-4d148b3791dc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:39:22.721315 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:22.721298    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtzzk\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-kube-api-access-qtzzk\") pod \"prometheus-k8s-0\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:23.171401 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:23.171369    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:23.171585 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:23.171423    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:23.171585 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:23.171452    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:23.171585 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:23.171491    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:23.171585 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.171500    6014 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 18:39:23.171585 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:23.171531    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:23.171585 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.171576    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:24.171560968 +0000 UTC m=+2.814728573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-config" not registered
Oct 30 18:39:23.171585 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.171580    6014 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 18:39:23.171896 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.171590    6014 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 18:39:23.171896 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:23.171596    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:23.171896 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.171603    6014 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 18:39:23.171896 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.171503    6014 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 18:39:23.171896 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.171646    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:24.17163216 +0000 UTC m=+2.814799752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 18:39:23.171896 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.171668    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:24.171655575 +0000 UTC m=+2.814823178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 18:39:23.171896 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.171681    6014 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 18:39:23.171896 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.171689    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:24.17167723 +0000 UTC m=+2.814844826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"service-ca" not registered
Oct 30 18:39:23.171896 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.171702    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:24.171696802 +0000 UTC m=+2.814864394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 18:39:23.171896 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.171712    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:24.171707151 +0000 UTC m=+2.814874743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 18:39:23.257016 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.256983    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:39:23.272496 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:23.272469    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpn9\" (UniqueName: \"kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:23.272672 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.272651    6014 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 18:39:23.272731 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.272675    6014 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 18:39:23.272731 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.272688    6014 projected.go:196] Error preparing data for projected volume kube-api-access-7lpn9 for pod openshift-console/console-6bbbd6864b-w27vc: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 18:39:23.272822 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:23.272775    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9 podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:24.272753233 +0000 UTC m=+2.915920831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lpn9" (UniqueName: "kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 18:39:23.639594 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:23.639565    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:24.181263 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:24.181229    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:24.181444 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:24.181320    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:24.181444 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.181354    6014 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 18:39:24.181444 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:24.181376    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:24.181444 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.181426    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:26.181407092 +0000 UTC m=+4.824574687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 18:39:24.181444 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.181430    6014 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 18:39:24.181733 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.181463    6014 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 18:39:24.181733 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.181467    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:26.181457126 +0000 UTC m=+4.824624745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-config" not registered
Oct 30 18:39:24.181733 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:24.181525    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:24.181733 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:24.181572    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:24.181733 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.181607    6014 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 18:39:24.181733 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.181619    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:26.181601237 +0000 UTC m=+4.824768831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 18:39:24.181733 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.181651    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:26.181640943 +0000 UTC m=+4.824808548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 18:39:24.181733 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.181665    6014 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 18:39:24.181733 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:24.181667    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:24.181733 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.181704    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:26.181691114 +0000 UTC m=+4.824858724 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 18:39:24.182016 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.181762    6014 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 18:39:24.182016 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.181794    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:26.181782416 +0000 UTC m=+4.824950035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"service-ca" not registered
Oct 30 18:39:24.254438 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:24.254406    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4kspp"
Oct 30 18:39:24.283109 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:24.283068    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpn9\" (UniqueName: \"kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:24.283313 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.283236    6014 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 18:39:24.283313 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.283253    6014 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 18:39:24.283313 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.283261    6014 projected.go:196] Error preparing data for projected volume kube-api-access-7lpn9 for pod openshift-console/console-6bbbd6864b-w27vc: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 18:39:24.283313 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:24.283312    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9 podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:26.283298876 +0000 UTC m=+4.926466490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lpn9" (UniqueName: "kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 18:39:25.653059 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:25.653029    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:26.206200 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:26.206128    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:26.206407 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:26.206222    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:26.206407 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:26.206244    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:26.206407 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.206266    6014 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 18:39:26.206407 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.206334    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:30.206319732 +0000 UTC m=+8.849487352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-config" not registered
Oct 30 18:39:26.206407 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.206351    6014 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 18:39:26.206407 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.206353    6014 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 18:39:26.206407 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:26.206270    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:26.206407 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.206406    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:30.20639236 +0000 UTC m=+8.849559955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 18:39:26.206697 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.206423    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:30.206417227 +0000 UTC m=+8.849584819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 18:39:26.206697 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.206427    6014 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 18:39:26.206697 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:26.206455    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:26.206697 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.206474    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:30.206461738 +0000 UTC m=+8.849629348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 18:39:26.206697 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.206488    6014 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 18:39:26.206697 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.206513    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:30.20650684 +0000 UTC m=+8.849674432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"service-ca" not registered
Oct 30 18:39:26.206697 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:26.206527    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:26.206697 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.206579    6014 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 18:39:26.206697 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.206602    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:30.206594796 +0000 UTC m=+8.849762402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 18:39:26.307457 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:26.307418    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpn9\" (UniqueName: \"kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:26.307611 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.307582    6014 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 18:39:26.307611 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.307604    6014 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 18:39:26.307698 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.307615    6014 projected.go:196] Error preparing data for projected volume kube-api-access-7lpn9 for pod openshift-console/console-6bbbd6864b-w27vc: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 18:39:26.307698 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:26.307670    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9 podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:30.307657357 +0000 UTC m=+8.950824964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lpn9" (UniqueName: "kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 18:39:26.761119 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:26.761086    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-68b7f4fd8f-mttl7"
Oct 30 18:39:27.370538 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:27.370504    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wwvr8"
Oct 30 18:39:28.391235 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:28.391197    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:39:30.252261 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:30.252213    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:30.252293    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:30.252323    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.252345    6014 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.252402    6014 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.252426    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:38.252409938 +0000 UTC m=+16.895577533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-config" not registered
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.252436    6014 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.252450    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:38.252438445 +0000 UTC m=+16.895606038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:30.252478    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.252494    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:38.252481114 +0000 UTC m=+16.895648723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.252528    6014 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:30.252547    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.252552    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:38.252545936 +0000 UTC m=+16.895713527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.252575    6014 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.252595    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:38.252589124 +0000 UTC m=+16.895756716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"service-ca" not registered
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:30.252606    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.252700    6014 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 18:39:30.252762 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.252745    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:38.252735627 +0000 UTC m=+16.895903238 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 18:39:30.353462 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:30.353421    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpn9\" (UniqueName: \"kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:30.353621 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.353558    6014 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 18:39:30.353621 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.353572    6014 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 18:39:30.353621 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.353581    6014 projected.go:196] Error preparing data for projected volume kube-api-access-7lpn9 for pod openshift-console/console-6bbbd6864b-w27vc: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 18:39:30.353753 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:30.353637    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9 podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:38.353621568 +0000 UTC m=+16.996789177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lpn9" (UniqueName: "kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 18:39:30.373943 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:30.373916    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:30.374164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:30.374134    6014 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:39:31.913991 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:31.913953    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gbbdc"
Oct 30 18:39:38.334972 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:38.334931    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:38.335002    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:38.335045    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.335086    6014 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:38.335115    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.335121    6014 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.335199    6014 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:38.335208    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.335218    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:54.335201719 +0000 UTC m=+32.978369324 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.335238    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:54.335230909 +0000 UTC m=+32.978398500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"service-ca" not registered
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.335261    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:54.335248147 +0000 UTC m=+32.978415740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.335263    6014 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:38.335289    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.335310    6014 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.335371    6014 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.335320    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:54.335305211 +0000 UTC m=+32.978472822 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-config" not registered
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.335408    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:54.335401226 +0000 UTC m=+32.978568845 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 18:39:38.335623 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.335420    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:54.335414794 +0000 UTC m=+32.978582385 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 18:39:38.436113 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:38.436079    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpn9\" (UniqueName: \"kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9\") pod \"console-6bbbd6864b-w27vc\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") " pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:38.436319 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.436297    6014 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 18:39:38.436387 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.436329    6014 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 18:39:38.436387 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.436343    6014 projected.go:196] Error preparing data for projected volume kube-api-access-7lpn9 for pod openshift-console/console-6bbbd6864b-w27vc: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 18:39:38.436490 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:38.436424    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9 podName:73dec725-685b-4a6d-81b4-783f6c79494e nodeName:}" failed. No retries permitted until 2026-10-30 18:39:54.43640319 +0000 UTC m=+33.079570805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lpn9" (UniqueName: "kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9") pod "console-6bbbd6864b-w27vc" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 18:39:42.618590 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:42.618560    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7848bc9d85-wmsb2"
Oct 30 18:39:47.558816 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.558771    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6bbbd6864b-w27vc" podUID="73dec725-685b-4a6d-81b4-783f6c79494e" containerName="console" containerID="cri-o://84fccaea4bf5830aa95561b37f58f0989d173a527f1b7b83830fd475446883bd" gracePeriod=15
Oct 30 18:39:47.810620 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.810564    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:47.827856 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.827835    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca\") pod \"73dec725-685b-4a6d-81b4-783f6c79494e\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") "
Oct 30 18:39:47.827957 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.827867    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config\") pod \"73dec725-685b-4a6d-81b4-783f6c79494e\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") "
Oct 30 18:39:47.827957 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.827895    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert\") pod \"73dec725-685b-4a6d-81b4-783f6c79494e\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") "
Oct 30 18:39:47.828040 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.827963    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle\") pod \"73dec725-685b-4a6d-81b4-783f6c79494e\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") "
Oct 30 18:39:47.828040 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.827995    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9\" (UniqueName: \"kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9\") pod \"73dec725-685b-4a6d-81b4-783f6c79494e\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") "
Oct 30 18:39:47.828040 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.828029    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dec725-685b-4a6d-81b4-783f6c79494e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/73dec725-685b-4a6d-81b4-783f6c79494e-tmp\") pod \"73dec725-685b-4a6d-81b4-783f6c79494e\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") "
Oct 30 18:39:47.828197 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.828051    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert\") pod \"73dec725-685b-4a6d-81b4-783f6c79494e\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") "
Oct 30 18:39:47.828197 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.828077    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config\") pod \"73dec725-685b-4a6d-81b4-783f6c79494e\" (UID: \"73dec725-685b-4a6d-81b4-783f6c79494e\") "
Oct 30 18:39:47.828400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.828314    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca" pod "73dec725-685b-4a6d-81b4-783f6c79494e" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:47.828400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.828335    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dec725-685b-4a6d-81b4-783f6c79494e-tmp" pod "73dec725-685b-4a6d-81b4-783f6c79494e" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:47.828549 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.828396    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle" pod "73dec725-685b-4a6d-81b4-783f6c79494e" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:47.828608 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.828574    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert" pod "73dec725-685b-4a6d-81b4-783f6c79494e" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:47.828666 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.828634    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config" pod "73dec725-685b-4a6d-81b4-783f6c79494e" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:47.830362 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.830341    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert" pod "73dec725-685b-4a6d-81b4-783f6c79494e" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:47.830473 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.830454    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config" pod "73dec725-685b-4a6d-81b4-783f6c79494e" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:47.830512 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.830484    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9" pod "73dec725-685b-4a6d-81b4-783f6c79494e" (UID: "73dec725-685b-4a6d-81b4-783f6c79494e"). InnerVolumeSpecName "kube-api-access-7lpn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:47.929438 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.929409    6014 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-trusted-ca-bundle\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:47.929438 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.929436    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lpn9\" (UniqueName: \"kubernetes.io/projected/73dec725-685b-4a6d-81b4-783f6c79494e-kube-api-access-7lpn9\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:47.929623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.929447    6014 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73dec725-685b-4a6d-81b4-783f6c79494e-tmp\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:47.929623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.929457    6014 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-oauth-serving-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:47.929623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.929466    6014 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-console-config\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:47.929623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.929474    6014 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73dec725-685b-4a6d-81b4-783f6c79494e-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:47.929623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.929483    6014 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-oauth-config\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:47.929623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:47.929491    6014 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dec725-685b-4a6d-81b4-783f6c79494e-console-serving-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:48.643101 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:48.643070    6014 generic.go:356] "Generic (PLEG): container finished" podID="73dec725-685b-4a6d-81b4-783f6c79494e" containerID="84fccaea4bf5830aa95561b37f58f0989d173a527f1b7b83830fd475446883bd" exitCode=0
Oct 30 18:39:48.643548 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:48.643117    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbbd6864b-w27vc"
Oct 30 18:39:48.643548 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:48.643166    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbbd6864b-w27vc" event={"ID":"73dec725-685b-4a6d-81b4-783f6c79494e","Type":"ContainerDied","Data":"84fccaea4bf5830aa95561b37f58f0989d173a527f1b7b83830fd475446883bd"}
Oct 30 18:39:48.643548 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:48.643220    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbbd6864b-w27vc" event={"ID":"73dec725-685b-4a6d-81b4-783f6c79494e","Type":"ContainerDied","Data":"80f03a6c2a94341aabe2204002e5b372c379b9f548b9d7e725600f186df1106c"}
Oct 30 18:39:48.643548 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:48.643247    6014 scope.go:122] "RemoveContainer" containerID="84fccaea4bf5830aa95561b37f58f0989d173a527f1b7b83830fd475446883bd"
Oct 30 18:39:48.652004 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:48.651982    6014 scope.go:122] "RemoveContainer" containerID="84fccaea4bf5830aa95561b37f58f0989d173a527f1b7b83830fd475446883bd"
Oct 30 18:39:48.652301 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:48.652284    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84fccaea4bf5830aa95561b37f58f0989d173a527f1b7b83830fd475446883bd\": container with ID starting with 84fccaea4bf5830aa95561b37f58f0989d173a527f1b7b83830fd475446883bd not found: ID does not exist" containerID="84fccaea4bf5830aa95561b37f58f0989d173a527f1b7b83830fd475446883bd"
Oct 30 18:39:48.652356 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:48.652309    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84fccaea4bf5830aa95561b37f58f0989d173a527f1b7b83830fd475446883bd"} err="failed to get container status \"84fccaea4bf5830aa95561b37f58f0989d173a527f1b7b83830fd475446883bd\": rpc error: code = NotFound desc = could not find container \"84fccaea4bf5830aa95561b37f58f0989d173a527f1b7b83830fd475446883bd\": container with ID starting with 84fccaea4bf5830aa95561b37f58f0989d173a527f1b7b83830fd475446883bd not found: ID does not exist"
Oct 30 18:39:48.804040 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:48.804007    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bbbd6864b-w27vc"]
Oct 30 18:39:48.853346 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:48.853308    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bbbd6864b-w27vc"]
Oct 30 18:39:49.473585 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:49.473544    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73dec725-685b-4a6d-81b4-783f6c79494e" path="/var/lib/kubelet/pods/73dec725-685b-4a6d-81b4-783f6c79494e/volumes"
Oct 30 18:39:51.609696 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:51.609663    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:39:51.610128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:51.610048    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="alertmanager" containerID="cri-o://273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960" gracePeriod=120
Oct 30 18:39:51.610248 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:51.610091    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="kube-rbac-proxy-metric" containerID="cri-o://90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f" gracePeriod=120
Oct 30 18:39:51.610248 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:51.610170    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="config-reloader" containerID="cri-o://e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888" gracePeriod=120
Oct 30 18:39:51.610248 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:51.610162    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="kube-rbac-proxy" containerID="cri-o://49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95" gracePeriod=120
Oct 30 18:39:51.610248 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:51.610106    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="kube-rbac-proxy-web" containerID="cri-o://873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8" gracePeriod=120
Oct 30 18:39:51.610450 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:51.610299    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="prom-label-proxy" containerID="cri-o://6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e" gracePeriod=120
Oct 30 18:39:52.324755 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.324728    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54qsj"
Oct 30 18:39:52.661677 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.661638    6014 generic.go:356] "Generic (PLEG): container finished" podID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerID="6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e" exitCode=0
Oct 30 18:39:52.661677 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.661669    6014 generic.go:356] "Generic (PLEG): container finished" podID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerID="49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95" exitCode=0
Oct 30 18:39:52.661677 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.661680    6014 generic.go:356] "Generic (PLEG): container finished" podID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerID="e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888" exitCode=0
Oct 30 18:39:52.662084 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.661689    6014 generic.go:356] "Generic (PLEG): container finished" podID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerID="273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960" exitCode=0
Oct 30 18:39:52.662084 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.661704    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerDied","Data":"6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e"}
Oct 30 18:39:52.662084 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.661771    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerDied","Data":"49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95"}
Oct 30 18:39:52.662084 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.661797    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerDied","Data":"e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888"}
Oct 30 18:39:52.662084 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.661822    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerDied","Data":"273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960"}
Oct 30 18:39:52.896043 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.895980    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:52.971076 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.971048    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-tls-assets\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-tls-assets\") pod \"e5f54b22-ebcb-43e3-8a45-d61242595fca\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") "
Oct 30 18:39:52.971270 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.971105    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-kube-api-access-bf5zd\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-kube-api-access-bf5zd\") pod \"e5f54b22-ebcb-43e3-8a45-d61242595fca\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") "
Oct 30 18:39:52.971270 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.971166    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e5f54b22-ebcb-43e3-8a45-d61242595fca\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") "
Oct 30 18:39:52.971270 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.971203    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy\") pod \"e5f54b22-ebcb-43e3-8a45-d61242595fca\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") "
Oct 30 18:39:52.971270 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.971222    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-volume\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-volume\") pod \"e5f54b22-ebcb-43e3-8a45-d61242595fca\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") "
Oct 30 18:39:52.971270 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.971257    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-main-tls\") pod \"e5f54b22-ebcb-43e3-8a45-d61242595fca\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") "
Oct 30 18:39:52.971533 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.971298    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-metrics-client-ca\") pod \"e5f54b22-ebcb-43e3-8a45-d61242595fca\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") "
Oct 30 18:39:52.971533 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.971337    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-trusted-ca-bundle\") pod \"e5f54b22-ebcb-43e3-8a45-d61242595fca\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") "
Oct 30 18:39:52.971533 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.971387    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-cluster-tls-config\") pod \"e5f54b22-ebcb-43e3-8a45-d61242595fca\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") "
Oct 30 18:39:52.971533 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.971418    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-web-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-web-config\") pod \"e5f54b22-ebcb-43e3-8a45-d61242595fca\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") "
Oct 30 18:39:52.971533 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.971447    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-out\") pod \"e5f54b22-ebcb-43e3-8a45-d61242595fca\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") "
Oct 30 18:39:52.971533 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.971486    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-main-db\") pod \"e5f54b22-ebcb-43e3-8a45-d61242595fca\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") "
Oct 30 18:39:52.971850 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.971534    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-web\") pod \"e5f54b22-ebcb-43e3-8a45-d61242595fca\" (UID: \"e5f54b22-ebcb-43e3-8a45-d61242595fca\") "
Oct 30 18:39:52.973021 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.972769    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-metrics-client-ca" pod "e5f54b22-ebcb-43e3-8a45-d61242595fca" (UID: "e5f54b22-ebcb-43e3-8a45-d61242595fca"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:52.973569 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.973316    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-trusted-ca-bundle" pod "e5f54b22-ebcb-43e3-8a45-d61242595fca" (UID: "e5f54b22-ebcb-43e3-8a45-d61242595fca"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:52.974235 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.974208    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-main-db" pod "e5f54b22-ebcb-43e3-8a45-d61242595fca" (UID: "e5f54b22-ebcb-43e3-8a45-d61242595fca"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:52.974678 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.974647    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-tls-assets" pod "e5f54b22-ebcb-43e3-8a45-d61242595fca" (UID: "e5f54b22-ebcb-43e3-8a45-d61242595fca"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:52.974895 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.974870    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-volume" pod "e5f54b22-ebcb-43e3-8a45-d61242595fca" (UID: "e5f54b22-ebcb-43e3-8a45-d61242595fca"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:52.974976 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.974873    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-kube-api-access-bf5zd" pod "e5f54b22-ebcb-43e3-8a45-d61242595fca" (UID: "e5f54b22-ebcb-43e3-8a45-d61242595fca"). InnerVolumeSpecName "kube-api-access-bf5zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:52.975110 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.975059    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-metric" pod "e5f54b22-ebcb-43e3-8a45-d61242595fca" (UID: "e5f54b22-ebcb-43e3-8a45-d61242595fca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:52.975399 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.975376    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy" pod "e5f54b22-ebcb-43e3-8a45-d61242595fca" (UID: "e5f54b22-ebcb-43e3-8a45-d61242595fca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:52.975582 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.975559    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-web" pod "e5f54b22-ebcb-43e3-8a45-d61242595fca" (UID: "e5f54b22-ebcb-43e3-8a45-d61242595fca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:52.975675 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.975642    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-main-tls" pod "e5f54b22-ebcb-43e3-8a45-d61242595fca" (UID: "e5f54b22-ebcb-43e3-8a45-d61242595fca"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:52.976298 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.976278    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-out" pod "e5f54b22-ebcb-43e3-8a45-d61242595fca" (UID: "e5f54b22-ebcb-43e3-8a45-d61242595fca"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:52.980115 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.980089    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-cluster-tls-config" pod "e5f54b22-ebcb-43e3-8a45-d61242595fca" (UID: "e5f54b22-ebcb-43e3-8a45-d61242595fca"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:52.988410 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:52.988389    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-web-config" pod "e5f54b22-ebcb-43e3-8a45-d61242595fca" (UID: "e5f54b22-ebcb-43e3-8a45-d61242595fca"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:53.072805 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.072771    6014 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-main-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:53.072805 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.072801    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-metrics-client-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:53.072805 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.072811    6014 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:53.073024 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.072823    6014 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-cluster-tls-config\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:53.073024 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.072832    6014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-web-config\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:53.073024 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.072841    6014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-out\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:53.073024 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.072850    6014 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5f54b22-ebcb-43e3-8a45-d61242595fca-alertmanager-main-db\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:53.073024 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.072859    6014 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:53.073024 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.072869    6014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-tls-assets\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:53.073024 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.072877    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf5zd\" (UniqueName: \"kubernetes.io/projected/e5f54b22-ebcb-43e3-8a45-d61242595fca-kube-api-access-bf5zd\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:53.073024 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.072885    6014 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:53.073024 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.072895    6014 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:53.073024 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.072903    6014 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5f54b22-ebcb-43e3-8a45-d61242595fca-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:53.256860 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:53.256779    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:39:53.668847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.668807    6014 generic.go:356] "Generic (PLEG): container finished" podID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerID="90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f" exitCode=0
Oct 30 18:39:53.668847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.668832    6014 generic.go:356] "Generic (PLEG): container finished" podID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerID="873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8" exitCode=0
Oct 30 18:39:53.669392 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.668887    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerDied","Data":"90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f"}
Oct 30 18:39:53.669392 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.668918    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.669392 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.668975    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerDied","Data":"873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8"}
Oct 30 18:39:53.669392 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.669013    6014 scope.go:122] "RemoveContainer" containerID="6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e"
Oct 30 18:39:53.669392 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.669017    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f54b22-ebcb-43e3-8a45-d61242595fca","Type":"ContainerDied","Data":"cf8f58ce264a062b7151255601a6f5aab15ef00cc42e399e302fa0d279e113f9"}
Oct 30 18:39:53.683775 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.683745    6014 scope.go:122] "RemoveContainer" containerID="90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f"
Oct 30 18:39:53.691015 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.690999    6014 scope.go:122] "RemoveContainer" containerID="49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95"
Oct 30 18:39:53.701068 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.701046    6014 scope.go:122] "RemoveContainer" containerID="873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8"
Oct 30 18:39:53.707685 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.707668    6014 scope.go:122] "RemoveContainer" containerID="e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888"
Oct 30 18:39:53.714908 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.714890    6014 scope.go:122] "RemoveContainer" containerID="273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960"
Oct 30 18:39:53.721987 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.721972    6014 scope.go:122] "RemoveContainer" containerID="6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133"
Oct 30 18:39:53.728665 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.728648    6014 scope.go:122] "RemoveContainer" containerID="6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e"
Oct 30 18:39:53.728920 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:53.728901    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e\": container with ID starting with 6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e not found: ID does not exist" containerID="6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e"
Oct 30 18:39:53.728987 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.728933    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e"} err="failed to get container status \"6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e\": rpc error: code = NotFound desc = could not find container \"6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e\": container with ID starting with 6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e not found: ID does not exist"
Oct 30 18:39:53.728987 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.728954    6014 scope.go:122] "RemoveContainer" containerID="90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f"
Oct 30 18:39:53.729242 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:53.729223    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f\": container with ID starting with 90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f not found: ID does not exist" containerID="90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f"
Oct 30 18:39:53.729299 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.729247    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f"} err="failed to get container status \"90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f\": rpc error: code = NotFound desc = could not find container \"90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f\": container with ID starting with 90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f not found: ID does not exist"
Oct 30 18:39:53.729299 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.729263    6014 scope.go:122] "RemoveContainer" containerID="49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95"
Oct 30 18:39:53.729508 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:53.729492    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95\": container with ID starting with 49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95 not found: ID does not exist" containerID="49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95"
Oct 30 18:39:53.729551 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.729513    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95"} err="failed to get container status \"49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95\": rpc error: code = NotFound desc = could not find container \"49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95\": container with ID starting with 49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95 not found: ID does not exist"
Oct 30 18:39:53.729551 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.729530    6014 scope.go:122] "RemoveContainer" containerID="873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8"
Oct 30 18:39:53.729763 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:53.729749    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8\": container with ID starting with 873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8 not found: ID does not exist" containerID="873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8"
Oct 30 18:39:53.729814 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.729766    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8"} err="failed to get container status \"873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8\": rpc error: code = NotFound desc = could not find container \"873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8\": container with ID starting with 873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8 not found: ID does not exist"
Oct 30 18:39:53.729814 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.729784    6014 scope.go:122] "RemoveContainer" containerID="e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888"
Oct 30 18:39:53.729988 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:53.729971    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888\": container with ID starting with e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888 not found: ID does not exist" containerID="e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888"
Oct 30 18:39:53.730025 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.729993    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888"} err="failed to get container status \"e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888\": rpc error: code = NotFound desc = could not find container \"e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888\": container with ID starting with e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888 not found: ID does not exist"
Oct 30 18:39:53.730059 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.730028    6014 scope.go:122] "RemoveContainer" containerID="273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960"
Oct 30 18:39:53.730313 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:53.730297    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960\": container with ID starting with 273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960 not found: ID does not exist" containerID="273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960"
Oct 30 18:39:53.730361 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.730317    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960"} err="failed to get container status \"273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960\": rpc error: code = NotFound desc = could not find container \"273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960\": container with ID starting with 273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960 not found: ID does not exist"
Oct 30 18:39:53.730361 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.730338    6014 scope.go:122] "RemoveContainer" containerID="6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133"
Oct 30 18:39:53.730583 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:53.730567    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133\": container with ID starting with 6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133 not found: ID does not exist" containerID="6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133"
Oct 30 18:39:53.730631 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.730587    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133"} err="failed to get container status \"6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133\": rpc error: code = NotFound desc = could not find container \"6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133\": container with ID starting with 6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133 not found: ID does not exist"
Oct 30 18:39:53.730631 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.730600    6014 scope.go:122] "RemoveContainer" containerID="6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e"
Oct 30 18:39:53.730817 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.730801    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e"} err="failed to get container status \"6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e\": rpc error: code = NotFound desc = could not find container \"6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e\": container with ID starting with 6b82991bf3de25afc8909bc026c86d33a6842ff50a38ff2e2a2b28bb1651f51e not found: ID does not exist"
Oct 30 18:39:53.730867 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.730817    6014 scope.go:122] "RemoveContainer" containerID="90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f"
Oct 30 18:39:53.731042 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.731026    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f"} err="failed to get container status \"90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f\": rpc error: code = NotFound desc = could not find container \"90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f\": container with ID starting with 90a7e72ce8c1e24fae9e1f62f39351e2fbf0458a84170742e27dcb41f191924f not found: ID does not exist"
Oct 30 18:39:53.731042 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.731041    6014 scope.go:122] "RemoveContainer" containerID="49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95"
Oct 30 18:39:53.731258 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.731243    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95"} err="failed to get container status \"49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95\": rpc error: code = NotFound desc = could not find container \"49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95\": container with ID starting with 49a87fceb1dd5f334007cf6abdb6b4e143536a353d5002ad2752b17b1782dd95 not found: ID does not exist"
Oct 30 18:39:53.731258 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.731258    6014 scope.go:122] "RemoveContainer" containerID="873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8"
Oct 30 18:39:53.731483 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.731467    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8"} err="failed to get container status \"873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8\": rpc error: code = NotFound desc = could not find container \"873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8\": container with ID starting with 873b220cba1c98e17677f996a8eee5aec9a8ddfec92f8eef47134ef05fce4ac8 not found: ID does not exist"
Oct 30 18:39:53.731529 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.731483    6014 scope.go:122] "RemoveContainer" containerID="e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888"
Oct 30 18:39:53.731697 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.731677    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888"} err="failed to get container status \"e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888\": rpc error: code = NotFound desc = could not find container \"e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888\": container with ID starting with e135b9025a4760ff88804169a74f6f0fa7d0aeae4b7ee47c62f721050bd77888 not found: ID does not exist"
Oct 30 18:39:53.731754 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.731698    6014 scope.go:122] "RemoveContainer" containerID="273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960"
Oct 30 18:39:53.731912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.731895    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960"} err="failed to get container status \"273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960\": rpc error: code = NotFound desc = could not find container \"273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960\": container with ID starting with 273d1e1f517e9cdfe59db5d750841a9d512de03fef345a9e0b8135cc30133960 not found: ID does not exist"
Oct 30 18:39:53.731912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.731911    6014 scope.go:122] "RemoveContainer" containerID="6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133"
Oct 30 18:39:53.732170 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.732124    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133"} err="failed to get container status \"6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133\": rpc error: code = NotFound desc = could not find container \"6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133\": container with ID starting with 6014e83f77bdc29bf4eb8011390fa46ea96f945ddffa6be77b8e258c8a34e133 not found: ID does not exist"
Oct 30 18:39:53.883587 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.883558    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:39:53.933326 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.933252    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:39:53.933688 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.933669    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:39:53.934098 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.933986    6014 cpu_manager.go:414] "containerMap: removing container" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="init-config-reloader"
Oct 30 18:39:53.934237 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934195    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="init-config-reloader"
Oct 30 18:39:53.934237 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934225    6014 cpu_manager.go:414] "containerMap: removing container" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="prom-label-proxy"
Oct 30 18:39:53.934237 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934232    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="prom-label-proxy"
Oct 30 18:39:53.934360 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934241    6014 cpu_manager.go:414] "containerMap: removing container" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="alertmanager"
Oct 30 18:39:53.934360 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934247    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="alertmanager"
Oct 30 18:39:53.934360 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934255    6014 cpu_manager.go:414] "containerMap: removing container" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="kube-rbac-proxy-web"
Oct 30 18:39:53.934360 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934260    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="kube-rbac-proxy-web"
Oct 30 18:39:53.934360 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934271    6014 cpu_manager.go:414] "containerMap: removing container" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="kube-rbac-proxy-metric"
Oct 30 18:39:53.934360 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934276    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="kube-rbac-proxy-metric"
Oct 30 18:39:53.934360 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934290    6014 cpu_manager.go:414] "containerMap: removing container" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="config-reloader"
Oct 30 18:39:53.934360 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934295    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="config-reloader"
Oct 30 18:39:53.934360 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934304    6014 cpu_manager.go:414] "containerMap: removing container" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="kube-rbac-proxy"
Oct 30 18:39:53.934360 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934308    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="kube-rbac-proxy"
Oct 30 18:39:53.934360 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934319    6014 cpu_manager.go:414] "containerMap: removing container" podUID="73dec725-685b-4a6d-81b4-783f6c79494e" containerName="console"
Oct 30 18:39:53.934360 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934323    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73dec725-685b-4a6d-81b4-783f6c79494e" containerName="console"
Oct 30 18:39:53.934729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934388    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="kube-rbac-proxy-metric"
Oct 30 18:39:53.934729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934396    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="config-reloader"
Oct 30 18:39:53.934729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934401    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="kube-rbac-proxy"
Oct 30 18:39:53.934729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934408    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="73dec725-685b-4a6d-81b4-783f6c79494e" containerName="console"
Oct 30 18:39:53.934729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934416    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="init-config-reloader"
Oct 30 18:39:53.934729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934421    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="prom-label-proxy"
Oct 30 18:39:53.934729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934427    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="alertmanager"
Oct 30 18:39:53.934729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.934432    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" containerName="kube-rbac-proxy-web"
Oct 30 18:39:53.956932 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.956909    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.980196 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.980161    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1156af79-a794-494b-98aa-28f8f0f42e58-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.980313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.980214    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.980313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.980263    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpqt9\" (UniqueName: \"kubernetes.io/projected/1156af79-a794-494b-98aa-28f8f0f42e58-kube-api-access-gpqt9\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.980380 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.980314    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1156af79-a794-494b-98aa-28f8f0f42e58-config-out\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.980413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.980387    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-web-config\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.980448 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.980417    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.980481 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.980449    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1156af79-a794-494b-98aa-28f8f0f42e58-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.980512 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.980486    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.980573 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.980555    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.980604 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.980588    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1156af79-a794-494b-98aa-28f8f0f42e58-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.980636 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.980607    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.980674 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.980663    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1156af79-a794-494b-98aa-28f8f0f42e58-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:53.980720 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:53.980704    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-config-volume\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.009720 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.009648    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 18:39:54.009720 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.009690    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 18:39:54.009720 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.009705    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 18:39:54.009720 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.009705    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 18:39:54.010026 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.009744    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 18:39:54.010026 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.009753    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 18:39:54.010026 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.009931    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 18:39:54.010026 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.009936    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 18:39:54.010026 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.009978    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bftj9\""
Oct 30 18:39:54.059341 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.059305    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 18:39:54.081676 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.081643    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-config-volume\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.081816 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.081684    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1156af79-a794-494b-98aa-28f8f0f42e58-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.081894 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.081813    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.081894 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.081850    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqt9\" (UniqueName: \"kubernetes.io/projected/1156af79-a794-494b-98aa-28f8f0f42e58-kube-api-access-gpqt9\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.081894 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.081884    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1156af79-a794-494b-98aa-28f8f0f42e58-config-out\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.082052 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.081909    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-web-config\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.082052 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.081936    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.082052 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.081986    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1156af79-a794-494b-98aa-28f8f0f42e58-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.082052 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.082012    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.082052 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.082048    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.082338 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.082081    6014 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:39:54.082338 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.082085    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1156af79-a794-494b-98aa-28f8f0f42e58-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.082338 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.082207    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.082338 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.082252    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1156af79-a794-494b-98aa-28f8f0f42e58-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.082589 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.082565    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1156af79-a794-494b-98aa-28f8f0f42e58-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.082656 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.082569    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1156af79-a794-494b-98aa-28f8f0f42e58-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.083736 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.083707    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1156af79-a794-494b-98aa-28f8f0f42e58-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.085182 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.085118    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1156af79-a794-494b-98aa-28f8f0f42e58-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.085916 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.085871    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-config-volume\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.085916 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.085899    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.086067 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.085989    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1156af79-a794-494b-98aa-28f8f0f42e58-config-out\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.086283 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.086262    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.086283 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.086275    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.086451 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.086434    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-web-config\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.086451 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.086441    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.087058 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.087042    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1156af79-a794-494b-98aa-28f8f0f42e58-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.138584 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.138556    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpqt9\" (UniqueName: \"kubernetes.io/projected/1156af79-a794-494b-98aa-28f8f0f42e58-kube-api-access-gpqt9\") pod \"alertmanager-main-0\" (UID: \"1156af79-a794-494b-98aa-28f8f0f42e58\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.169954 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.169921    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:39:54.386272 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.386235    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:54.892057 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:54.892026    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:39:54.895756 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:54.895730    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1156af79_a794_494b_98aa_28f8f0f42e58.slice/crio-dab4fef421e0097b43d7c7e4017d7bbc2547fdac57fc10ff6aae2aac46dc34c9 WatchSource:0}: Error finding container dab4fef421e0097b43d7c7e4017d7bbc2547fdac57fc10ff6aae2aac46dc34c9: Status 404 returned error can't find the container with id dab4fef421e0097b43d7c7e4017d7bbc2547fdac57fc10ff6aae2aac46dc34c9
Oct 30 18:39:55.473547 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:55.473516    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f54b22-ebcb-43e3-8a45-d61242595fca" path="/var/lib/kubelet/pods/e5f54b22-ebcb-43e3-8a45-d61242595fca/volumes"
Oct 30 18:39:55.681811 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:55.681781    6014 generic.go:356] "Generic (PLEG): container finished" podID="1156af79-a794-494b-98aa-28f8f0f42e58" containerID="bcc17fadbcb8412db7d8ab819edd5364bf2fefd92ba4d62991e1fb718e8c556b" exitCode=0
Oct 30 18:39:55.681946 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:55.681820    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1156af79-a794-494b-98aa-28f8f0f42e58","Type":"ContainerDied","Data":"bcc17fadbcb8412db7d8ab819edd5364bf2fefd92ba4d62991e1fb718e8c556b"}
Oct 30 18:39:55.681946 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:55.681872    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1156af79-a794-494b-98aa-28f8f0f42e58","Type":"ContainerStarted","Data":"dab4fef421e0097b43d7c7e4017d7bbc2547fdac57fc10ff6aae2aac46dc34c9"}
Oct 30 18:39:56.688594 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:56.688557    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1156af79-a794-494b-98aa-28f8f0f42e58","Type":"ContainerStarted","Data":"21313b22cc7a9731f6d1ae5ea134f371774b5f5f040ed3ecb894ffa9b4f1e410"}
Oct 30 18:39:56.688991 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:56.688636    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1156af79-a794-494b-98aa-28f8f0f42e58","Type":"ContainerStarted","Data":"3187dc2efb8b5167ccfbbcc57b29b8218add2b435f083de54ee8d767032e3a3e"}
Oct 30 18:39:56.688991 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:56.688676    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1156af79-a794-494b-98aa-28f8f0f42e58","Type":"ContainerStarted","Data":"da971407e35911e08ab77c01ae8fe07b4f83a1813f1ab432d9dcd943ea27bc94"}
Oct 30 18:39:56.688991 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:56.688715    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1156af79-a794-494b-98aa-28f8f0f42e58","Type":"ContainerStarted","Data":"5b99ffbf3ad5dde8db30633308c03a1cb7bc706832f3cf76560548719cd18124"}
Oct 30 18:39:56.688991 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:56.688751    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1156af79-a794-494b-98aa-28f8f0f42e58","Type":"ContainerStarted","Data":"57c869a6d770862b7467516361a775b9197a86afc68756c6d59d71a87a8355ab"}
Oct 30 18:39:56.688991 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:56.688786    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1156af79-a794-494b-98aa-28f8f0f42e58","Type":"ContainerStarted","Data":"1ef8946788c1fdeda9bacfe1514adb0059676ca26d321f596652aeb50347c276"}
Oct 30 18:39:56.810988 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:56.810938    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.8043016339999998 podStartE2EDuration="3.81092427s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.622636ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:56.810191239 +0000 UTC m=+35.453358854" watchObservedRunningTime="2026-10-30 18:39:56.81092427 +0000 UTC m=+35.454091880"
Oct 30 18:39:57.343275 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.343244    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:57.343582 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.343547    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="prometheus" containerID="cri-o://450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5" gracePeriod=600
Oct 30 18:39:57.343671 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.343573    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="config-reloader" containerID="cri-o://14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66" gracePeriod=600
Oct 30 18:39:57.343671 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.343612    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43" gracePeriod=600
Oct 30 18:39:57.343671 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.343615    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="kube-rbac-proxy-web" containerID="cri-o://f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66" gracePeriod=600
Oct 30 18:39:57.343671 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.343573    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="thanos-sidecar" containerID="cri-o://ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400" gracePeriod=600
Oct 30 18:39:57.343836 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.343547    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="kube-rbac-proxy" containerID="cri-o://24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed" gracePeriod=600
Oct 30 18:39:57.589038 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.589016    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.612934 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.612882    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-thanos-prometheus-http-client-file\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.612934 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.612919    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-serving-certs-ca-bundle\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.612942    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-2\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.612962    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-trusted-ca-bundle\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.612996    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-grpc-tls\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613052    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-kube-rbac-proxy\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613292 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613086    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-db\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613292 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613115    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-web-config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-web-config\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613292 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613183    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-0\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613292 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613213    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613292 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613257    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613292 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613290    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-kubelet-serving-ca-bundle\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613563 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613316    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-metrics-client-ca\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613563 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613353    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-2" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:57.613563 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613362    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-metrics-client-certs\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613563 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613397    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-config\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.613563 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613397    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-serving-certs-ca-bundle" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:57.613563 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613403    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-trusted-ca-bundle" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:57.613563 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.613432    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-tls\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.614358 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.614325    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-kubelet-serving-ca-bundle" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:57.614479 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.614423    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-1\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.614810 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.614788    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-tls-assets\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-tls-assets\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.614905 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.614848    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-config-out\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.614963 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.614897    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-kube-api-access-qtzzk\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-kube-api-access-qtzzk\") pod \"d615363f-bfb2-4a9f-8678-774d3effd7d6\" (UID: \"d615363f-bfb2-4a9f-8678-774d3effd7d6\") "
Oct 30 18:39:57.615609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.615583    6014 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.615696 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.615616    6014 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.615696 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.615640    6014 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.615801 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.615702    6014 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.615936 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.615905    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-metrics-client-ca" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:57.617060 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.616002    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-db" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:57.617172 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.617105    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-thanos-prometheus-http-client-file" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:57.618080 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.617708    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-1" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:57.618080 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.618040    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-0" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:57.619907 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.618496    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-grpc-tls" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:57.622502 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.622454    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-metrics-client-certs" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:57.622502 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.622476    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-config-out" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:57.622643 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.622562    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-kube-rbac-proxy-web" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:57.623247 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.622799    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-config" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:57.623463 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.623436    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-tls" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:57.623543 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.623503    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-kube-rbac-proxy" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:57.624078 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.623704    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-thanos-sidecar-tls" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:57.624189 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.624172    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-tls-assets" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:57.624752 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.624732    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-kube-api-access-qtzzk" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "kube-api-access-qtzzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:57.632592 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.632570    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-web-config" pod "d615363f-bfb2-4a9f-8678-774d3effd7d6" (UID: "d615363f-bfb2-4a9f-8678-774d3effd7d6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:57.696982 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.696945    6014 generic.go:356] "Generic (PLEG): container finished" podID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerID="abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43" exitCode=0
Oct 30 18:39:57.696982 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.696974    6014 generic.go:356] "Generic (PLEG): container finished" podID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerID="24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed" exitCode=0
Oct 30 18:39:57.697400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.696982    6014 generic.go:356] "Generic (PLEG): container finished" podID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerID="f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66" exitCode=0
Oct 30 18:39:57.697400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.696990    6014 generic.go:356] "Generic (PLEG): container finished" podID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerID="ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400" exitCode=0
Oct 30 18:39:57.697400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.696998    6014 generic.go:356] "Generic (PLEG): container finished" podID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerID="14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66" exitCode=0
Oct 30 18:39:57.697400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.697006    6014 generic.go:356] "Generic (PLEG): container finished" podID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerID="450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5" exitCode=0
Oct 30 18:39:57.697400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.697003    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerDied","Data":"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"}
Oct 30 18:39:57.697400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.697057    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.697400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.697104    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerDied","Data":"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"}
Oct 30 18:39:57.697400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.697163    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerDied","Data":"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"}
Oct 30 18:39:57.697400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.697169    6014 scope.go:122] "RemoveContainer" containerID="abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"
Oct 30 18:39:57.697400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.697194    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerDied","Data":"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"}
Oct 30 18:39:57.697400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.697230    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerDied","Data":"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"}
Oct 30 18:39:57.697400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.697269    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerDied","Data":"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"}
Oct 30 18:39:57.697400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.697306    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d615363f-bfb2-4a9f-8678-774d3effd7d6","Type":"ContainerDied","Data":"fe32fd6bee74600dc227871e74853cd90c58569364d9b319f3341099fe742d65"}
Oct 30 18:39:57.711045 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.711023    6014 scope.go:122] "RemoveContainer" containerID="24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"
Oct 30 18:39:57.716848 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.716823    6014 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-grpc-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.716943 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.716852    6014 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-kube-rbac-proxy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.716943 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.716867    6014 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-db\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.716943 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.716882    6014 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-web-config\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.716943 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.716896    6014 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.716943 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.716910    6014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.716943 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.716925    6014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.716943 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.716938    6014 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-configmap-metrics-client-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.717215 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.716951    6014 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-metrics-client-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.717215 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.716964    6014 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-config\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.717215 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.716975    6014 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.717215 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.716986    6014 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d615363f-bfb2-4a9f-8678-774d3effd7d6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.717215 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.716999    6014 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-tls-assets\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.717215 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.717011    6014 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d615363f-bfb2-4a9f-8678-774d3effd7d6-config-out\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.717215 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.717022    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtzzk\" (UniqueName: \"kubernetes.io/projected/d615363f-bfb2-4a9f-8678-774d3effd7d6-kube-api-access-qtzzk\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.717215 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.717033    6014 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d615363f-bfb2-4a9f-8678-774d3effd7d6-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:57.720235 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.720214    6014 scope.go:122] "RemoveContainer" containerID="f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"
Oct 30 18:39:57.728547 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.728527    6014 scope.go:122] "RemoveContainer" containerID="ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"
Oct 30 18:39:57.735836 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.735815    6014 scope.go:122] "RemoveContainer" containerID="14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"
Oct 30 18:39:57.743127 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.743105    6014 scope.go:122] "RemoveContainer" containerID="450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"
Oct 30 18:39:57.751753 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.751737    6014 scope.go:122] "RemoveContainer" containerID="004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"
Oct 30 18:39:57.758529 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.758509    6014 scope.go:122] "RemoveContainer" containerID="abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"
Oct 30 18:39:57.758812 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:57.758792    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43\": container with ID starting with abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43 not found: ID does not exist" containerID="abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"
Oct 30 18:39:57.758876 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.758823    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"} err="failed to get container status \"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43\": rpc error: code = NotFound desc = could not find container \"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43\": container with ID starting with abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43 not found: ID does not exist"
Oct 30 18:39:57.758876 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.758843    6014 scope.go:122] "RemoveContainer" containerID="24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"
Oct 30 18:39:57.759073 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:57.759056    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed\": container with ID starting with 24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed not found: ID does not exist" containerID="24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"
Oct 30 18:39:57.759164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.759083    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"} err="failed to get container status \"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed\": rpc error: code = NotFound desc = could not find container \"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed\": container with ID starting with 24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed not found: ID does not exist"
Oct 30 18:39:57.759164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.759104    6014 scope.go:122] "RemoveContainer" containerID="f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"
Oct 30 18:39:57.759559 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:57.759539    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66\": container with ID starting with f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66 not found: ID does not exist" containerID="f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"
Oct 30 18:39:57.759645 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.759562    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"} err="failed to get container status \"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66\": rpc error: code = NotFound desc = could not find container \"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66\": container with ID starting with f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66 not found: ID does not exist"
Oct 30 18:39:57.759645 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.759575    6014 scope.go:122] "RemoveContainer" containerID="ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"
Oct 30 18:39:57.759824 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:57.759806    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400\": container with ID starting with ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400 not found: ID does not exist" containerID="ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"
Oct 30 18:39:57.759878 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.759830    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"} err="failed to get container status \"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400\": rpc error: code = NotFound desc = could not find container \"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400\": container with ID starting with ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400 not found: ID does not exist"
Oct 30 18:39:57.759878 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.759844    6014 scope.go:122] "RemoveContainer" containerID="14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"
Oct 30 18:39:57.760063 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:57.760048    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66\": container with ID starting with 14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66 not found: ID does not exist" containerID="14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"
Oct 30 18:39:57.760120 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.760066    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"} err="failed to get container status \"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66\": rpc error: code = NotFound desc = could not find container \"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66\": container with ID starting with 14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66 not found: ID does not exist"
Oct 30 18:39:57.760120 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.760085    6014 scope.go:122] "RemoveContainer" containerID="450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"
Oct 30 18:39:57.760304 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:57.760281    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5\": container with ID starting with 450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5 not found: ID does not exist" containerID="450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"
Oct 30 18:39:57.760358 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.760333    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"} err="failed to get container status \"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5\": rpc error: code = NotFound desc = could not find container \"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5\": container with ID starting with 450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5 not found: ID does not exist"
Oct 30 18:39:57.760358 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.760346    6014 scope.go:122] "RemoveContainer" containerID="004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"
Oct 30 18:39:57.760554 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:39:57.760539    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced\": container with ID starting with 004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced not found: ID does not exist" containerID="004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"
Oct 30 18:39:57.760611 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.760555    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"} err="failed to get container status \"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced\": rpc error: code = NotFound desc = could not find container \"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced\": container with ID starting with 004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced not found: ID does not exist"
Oct 30 18:39:57.760611 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.760565    6014 scope.go:122] "RemoveContainer" containerID="abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"
Oct 30 18:39:57.760770 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.760751    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"} err="failed to get container status \"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43\": rpc error: code = NotFound desc = could not find container \"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43\": container with ID starting with abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43 not found: ID does not exist"
Oct 30 18:39:57.760833 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.760770    6014 scope.go:122] "RemoveContainer" containerID="24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"
Oct 30 18:39:57.761004 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.760984    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"} err="failed to get container status \"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed\": rpc error: code = NotFound desc = could not find container \"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed\": container with ID starting with 24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed not found: ID does not exist"
Oct 30 18:39:57.761085 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.761004    6014 scope.go:122] "RemoveContainer" containerID="f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"
Oct 30 18:39:57.761285 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.761267    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"} err="failed to get container status \"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66\": rpc error: code = NotFound desc = could not find container \"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66\": container with ID starting with f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66 not found: ID does not exist"
Oct 30 18:39:57.761361 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.761286    6014 scope.go:122] "RemoveContainer" containerID="ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"
Oct 30 18:39:57.761526 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.761506    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"} err="failed to get container status \"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400\": rpc error: code = NotFound desc = could not find container \"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400\": container with ID starting with ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400 not found: ID does not exist"
Oct 30 18:39:57.761585 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.761526    6014 scope.go:122] "RemoveContainer" containerID="14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"
Oct 30 18:39:57.761749 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.761732    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"} err="failed to get container status \"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66\": rpc error: code = NotFound desc = could not find container \"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66\": container with ID starting with 14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66 not found: ID does not exist"
Oct 30 18:39:57.761749 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.761749    6014 scope.go:122] "RemoveContainer" containerID="450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"
Oct 30 18:39:57.761979 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.761960    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"} err="failed to get container status \"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5\": rpc error: code = NotFound desc = could not find container \"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5\": container with ID starting with 450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5 not found: ID does not exist"
Oct 30 18:39:57.762036 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.761980    6014 scope.go:122] "RemoveContainer" containerID="004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"
Oct 30 18:39:57.762232 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.762215    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"} err="failed to get container status \"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced\": rpc error: code = NotFound desc = could not find container \"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced\": container with ID starting with 004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced not found: ID does not exist"
Oct 30 18:39:57.762298 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.762231    6014 scope.go:122] "RemoveContainer" containerID="abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"
Oct 30 18:39:57.762423 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.762406    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"} err="failed to get container status \"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43\": rpc error: code = NotFound desc = could not find container \"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43\": container with ID starting with abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43 not found: ID does not exist"
Oct 30 18:39:57.762482 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.762424    6014 scope.go:122] "RemoveContainer" containerID="24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"
Oct 30 18:39:57.762609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.762594    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"} err="failed to get container status \"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed\": rpc error: code = NotFound desc = could not find container \"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed\": container with ID starting with 24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed not found: ID does not exist"
Oct 30 18:39:57.762663 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.762609    6014 scope.go:122] "RemoveContainer" containerID="f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"
Oct 30 18:39:57.762807 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.762783    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"} err="failed to get container status \"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66\": rpc error: code = NotFound desc = could not find container \"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66\": container with ID starting with f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66 not found: ID does not exist"
Oct 30 18:39:57.762902 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.762809    6014 scope.go:122] "RemoveContainer" containerID="ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"
Oct 30 18:39:57.764617 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.764598    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"} err="failed to get container status \"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400\": rpc error: code = NotFound desc = could not find container \"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400\": container with ID starting with ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400 not found: ID does not exist"
Oct 30 18:39:57.764617 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.764615    6014 scope.go:122] "RemoveContainer" containerID="14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"
Oct 30 18:39:57.764826 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.764810    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"} err="failed to get container status \"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66\": rpc error: code = NotFound desc = could not find container \"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66\": container with ID starting with 14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66 not found: ID does not exist"
Oct 30 18:39:57.764874 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.764827    6014 scope.go:122] "RemoveContainer" containerID="450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"
Oct 30 18:39:57.765041 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.765026    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"} err="failed to get container status \"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5\": rpc error: code = NotFound desc = could not find container \"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5\": container with ID starting with 450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5 not found: ID does not exist"
Oct 30 18:39:57.765041 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.765041    6014 scope.go:122] "RemoveContainer" containerID="004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"
Oct 30 18:39:57.765279 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.765257    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"} err="failed to get container status \"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced\": rpc error: code = NotFound desc = could not find container \"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced\": container with ID starting with 004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced not found: ID does not exist"
Oct 30 18:39:57.765364 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.765281    6014 scope.go:122] "RemoveContainer" containerID="abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"
Oct 30 18:39:57.765469 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.765453    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"} err="failed to get container status \"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43\": rpc error: code = NotFound desc = could not find container \"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43\": container with ID starting with abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43 not found: ID does not exist"
Oct 30 18:39:57.765511 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.765469    6014 scope.go:122] "RemoveContainer" containerID="24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"
Oct 30 18:39:57.765726 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.765705    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"} err="failed to get container status \"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed\": rpc error: code = NotFound desc = could not find container \"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed\": container with ID starting with 24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed not found: ID does not exist"
Oct 30 18:39:57.765794 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.765726    6014 scope.go:122] "RemoveContainer" containerID="f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"
Oct 30 18:39:57.765969 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.765950    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"} err="failed to get container status \"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66\": rpc error: code = NotFound desc = could not find container \"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66\": container with ID starting with f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66 not found: ID does not exist"
Oct 30 18:39:57.765969 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.765966    6014 scope.go:122] "RemoveContainer" containerID="ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"
Oct 30 18:39:57.766234 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.766215    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"} err="failed to get container status \"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400\": rpc error: code = NotFound desc = could not find container \"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400\": container with ID starting with ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400 not found: ID does not exist"
Oct 30 18:39:57.766234 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.766234    6014 scope.go:122] "RemoveContainer" containerID="14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"
Oct 30 18:39:57.766467 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.766449    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"} err="failed to get container status \"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66\": rpc error: code = NotFound desc = could not find container \"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66\": container with ID starting with 14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66 not found: ID does not exist"
Oct 30 18:39:57.766533 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.766467    6014 scope.go:122] "RemoveContainer" containerID="450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"
Oct 30 18:39:57.766679 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.766656    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"} err="failed to get container status \"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5\": rpc error: code = NotFound desc = could not find container \"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5\": container with ID starting with 450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5 not found: ID does not exist"
Oct 30 18:39:57.766679 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.766673    6014 scope.go:122] "RemoveContainer" containerID="004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"
Oct 30 18:39:57.766874 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.766857    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"} err="failed to get container status \"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced\": rpc error: code = NotFound desc = could not find container \"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced\": container with ID starting with 004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced not found: ID does not exist"
Oct 30 18:39:57.766937 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.766874    6014 scope.go:122] "RemoveContainer" containerID="abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"
Oct 30 18:39:57.767059 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.767045    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"} err="failed to get container status \"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43\": rpc error: code = NotFound desc = could not find container \"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43\": container with ID starting with abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43 not found: ID does not exist"
Oct 30 18:39:57.767113 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.767059    6014 scope.go:122] "RemoveContainer" containerID="24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"
Oct 30 18:39:57.767276 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.767254    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"} err="failed to get container status \"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed\": rpc error: code = NotFound desc = could not find container \"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed\": container with ID starting with 24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed not found: ID does not exist"
Oct 30 18:39:57.767276 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.767274    6014 scope.go:122] "RemoveContainer" containerID="f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"
Oct 30 18:39:57.767517 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.767497    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"} err="failed to get container status \"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66\": rpc error: code = NotFound desc = could not find container \"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66\": container with ID starting with f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66 not found: ID does not exist"
Oct 30 18:39:57.767517 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.767515    6014 scope.go:122] "RemoveContainer" containerID="ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"
Oct 30 18:39:57.767741 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.767723    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"} err="failed to get container status \"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400\": rpc error: code = NotFound desc = could not find container \"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400\": container with ID starting with ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400 not found: ID does not exist"
Oct 30 18:39:57.767796 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.767741    6014 scope.go:122] "RemoveContainer" containerID="14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"
Oct 30 18:39:57.767987 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.767966    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"} err="failed to get container status \"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66\": rpc error: code = NotFound desc = could not find container \"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66\": container with ID starting with 14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66 not found: ID does not exist"
Oct 30 18:39:57.768047 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.767988    6014 scope.go:122] "RemoveContainer" containerID="450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"
Oct 30 18:39:57.768202 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.768186    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"} err="failed to get container status \"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5\": rpc error: code = NotFound desc = could not find container \"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5\": container with ID starting with 450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5 not found: ID does not exist"
Oct 30 18:39:57.768202 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.768201    6014 scope.go:122] "RemoveContainer" containerID="004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"
Oct 30 18:39:57.768423 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.768405    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"} err="failed to get container status \"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced\": rpc error: code = NotFound desc = could not find container \"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced\": container with ID starting with 004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced not found: ID does not exist"
Oct 30 18:39:57.768488 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.768424    6014 scope.go:122] "RemoveContainer" containerID="abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"
Oct 30 18:39:57.768628 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.768613    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43"} err="failed to get container status \"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43\": rpc error: code = NotFound desc = could not find container \"abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43\": container with ID starting with abfe1eb0c7de6116ee5ef78c603d7944b486255dd5b3fe0795cc3fa948303f43 not found: ID does not exist"
Oct 30 18:39:57.768682 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.768628    6014 scope.go:122] "RemoveContainer" containerID="24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"
Oct 30 18:39:57.768818 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.768801    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed"} err="failed to get container status \"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed\": rpc error: code = NotFound desc = could not find container \"24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed\": container with ID starting with 24feb0704646fdddd86abf767890a2085ad424db7b2218ef48970227f71bb8ed not found: ID does not exist"
Oct 30 18:39:57.768875 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.768818    6014 scope.go:122] "RemoveContainer" containerID="f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"
Oct 30 18:39:57.769051 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.769033    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66"} err="failed to get container status \"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66\": rpc error: code = NotFound desc = could not find container \"f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66\": container with ID starting with f46967ff792dc201b885e80f11686b9389179d9aa2d79e4ea097f00029d8dd66 not found: ID does not exist"
Oct 30 18:39:57.769110 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.769052    6014 scope.go:122] "RemoveContainer" containerID="ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"
Oct 30 18:39:57.769308 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.769290    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400"} err="failed to get container status \"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400\": rpc error: code = NotFound desc = could not find container \"ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400\": container with ID starting with ce4134a39e80a8c3a4959a7ac6d868e11bad9465e6197812575f03a95e235400 not found: ID does not exist"
Oct 30 18:39:57.769367 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.769308    6014 scope.go:122] "RemoveContainer" containerID="14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"
Oct 30 18:39:57.769517 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.769496    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66"} err="failed to get container status \"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66\": rpc error: code = NotFound desc = could not find container \"14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66\": container with ID starting with 14687cd168ab8a3c62f90e61576788ada42590ed0fddd59b6b5ec3a2447b7a66 not found: ID does not exist"
Oct 30 18:39:57.769517 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.769516    6014 scope.go:122] "RemoveContainer" containerID="450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"
Oct 30 18:39:57.769729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.769709    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5"} err="failed to get container status \"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5\": rpc error: code = NotFound desc = could not find container \"450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5\": container with ID starting with 450e51183655af0055ec683ca2b22dad644f5e2b239dce0243bc739ac28bd1d5 not found: ID does not exist"
Oct 30 18:39:57.769796 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.769731    6014 scope.go:122] "RemoveContainer" containerID="004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"
Oct 30 18:39:57.770026 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.770008    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced"} err="failed to get container status \"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced\": rpc error: code = NotFound desc = could not find container \"004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced\": container with ID starting with 004aec45c0c937471d85b73d5ccd4eb39df8a0e30038414af6e875f4b96f5ced not found: ID does not exist"
Oct 30 18:39:57.822254 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.822196    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:57.828437 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.828407    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:57.851456 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851426    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:57.851718 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851706    6014 cpu_manager.go:414] "containerMap: removing container" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="prometheus"
Oct 30 18:39:57.851755 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851719    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="prometheus"
Oct 30 18:39:57.851755 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851741    6014 cpu_manager.go:414] "containerMap: removing container" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="thanos-sidecar"
Oct 30 18:39:57.851755 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851746    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="thanos-sidecar"
Oct 30 18:39:57.851839 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851756    6014 cpu_manager.go:414] "containerMap: removing container" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="kube-rbac-proxy"
Oct 30 18:39:57.851839 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851761    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="kube-rbac-proxy"
Oct 30 18:39:57.851839 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851768    6014 cpu_manager.go:414] "containerMap: removing container" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="kube-rbac-proxy-web"
Oct 30 18:39:57.851839 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851774    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="kube-rbac-proxy-web"
Oct 30 18:39:57.851839 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851782    6014 cpu_manager.go:414] "containerMap: removing container" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="init-config-reloader"
Oct 30 18:39:57.851839 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851787    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="init-config-reloader"
Oct 30 18:39:57.851839 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851794    6014 cpu_manager.go:414] "containerMap: removing container" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="kube-rbac-proxy-thanos"
Oct 30 18:39:57.851839 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851799    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="kube-rbac-proxy-thanos"
Oct 30 18:39:57.851839 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851807    6014 cpu_manager.go:414] "containerMap: removing container" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="config-reloader"
Oct 30 18:39:57.851839 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851811    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="config-reloader"
Oct 30 18:39:57.852108 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851862    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="init-config-reloader"
Oct 30 18:39:57.852108 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851871    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="kube-rbac-proxy-thanos"
Oct 30 18:39:57.852108 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851877    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="config-reloader"
Oct 30 18:39:57.852108 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851884    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="prometheus"
Oct 30 18:39:57.852108 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851892    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="thanos-sidecar"
Oct 30 18:39:57.852108 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851898    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="kube-rbac-proxy"
Oct 30 18:39:57.852108 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.851902    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" containerName="kube-rbac-proxy-web"
Oct 30 18:39:57.855418 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.855403    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.907860 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.907722    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 18:39:57.908135 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.907969    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 18:39:57.908135 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.908175    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 18:39:57.908541 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.908180    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 18:39:57.908541 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.908308    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 18:39:57.908541 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.908330    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 18:39:57.908541 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.908331    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 18:39:57.908541 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.908335    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 18:39:57.908541 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.908357    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bdjh3qnjck7pd\""
Oct 30 18:39:57.908541 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.908412    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 18:39:57.909261 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.908654    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 18:39:57.909261 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.908665    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 18:39:57.909261 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.908825    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lx4z8\""
Oct 30 18:39:57.909261 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.908848    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 18:39:57.918492 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918471    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-config-out\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.918565 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918516    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.918565 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918545    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.918647 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918615    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.918680 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918651    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.918680 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918669    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.918778 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918684    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.918778 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918702    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.918863 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918779    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.918863 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918813    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.918863 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918833    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzlxr\" (UniqueName: \"kubernetes.io/projected/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-kube-api-access-jzlxr\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.919012 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918898    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.919012 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918926    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.919012 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.918995    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.919116 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.919025    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.919116 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.919058    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-config\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.919116 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.919112    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-web-config\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.919251 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.919130    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.919251 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.919173    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.919251 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.919198    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:57.958169 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.958123    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 18:39:57.958330 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.958174    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 18:39:57.958872 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:57.958852    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:58.020374 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020334    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.020374 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020373    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.020621 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020394    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.020621 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020422    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.020621 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020444    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.020621 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020468    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.020621 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020493    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.020621 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020524    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.020621 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020547    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.020621 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020569    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlxr\" (UniqueName: \"kubernetes.io/projected/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-kube-api-access-jzlxr\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.020621 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020602    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.021048 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020626    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.021048 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020679    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.021048 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020703    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.021048 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020727    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-config\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.021048 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020785    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-web-config\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.021048 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020809    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.021048 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020847    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.021048 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020872    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.021048 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.020903    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-config-out\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.024083 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.021072    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.024083 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.021248    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.024083 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.022658    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.024083 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.023221    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.024083 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.023757    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.024083 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.023954    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.024083 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.023985    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.024614 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.024198    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-config-out\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.024614 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.024583    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.024720 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.024619    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.024845 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.024826    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.025735 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.025382    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-web-config\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.025735 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.025673    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.025892 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.025805    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.025892 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.025886    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.026690 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.026663    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-config\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.026942 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.026925    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.027004 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.026978    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.027718 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.027693    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.079013 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.078981    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlxr\" (UniqueName: \"kubernetes.io/projected/5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff-kube-api-access-jzlxr\") pod \"prometheus-k8s-0\" (UID: \"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.171892 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.171802    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:58.393997 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.393968    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-27m75"
Oct 30 18:39:58.597885 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.597616    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:58.600240 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:39:58.600197    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f80fc66_0bc4_48c7_9f2f_2d63df3c5dff.slice/crio-e768772fd63b3fb379509ccbd6c1514d92d26bdb5b3e1eedaad3450073308bbb WatchSource:0}: Error finding container e768772fd63b3fb379509ccbd6c1514d92d26bdb5b3e1eedaad3450073308bbb: Status 404 returned error can't find the container with id e768772fd63b3fb379509ccbd6c1514d92d26bdb5b3e1eedaad3450073308bbb
Oct 30 18:39:58.703403 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.703365    6014 generic.go:356] "Generic (PLEG): container finished" podID="5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff" containerID="e8fdc81d2532cb558170c001e2a463dcd8e6eca6fd1448647113b3a387802930" exitCode=0
Oct 30 18:39:58.703733 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.703444    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff","Type":"ContainerDied","Data":"e8fdc81d2532cb558170c001e2a463dcd8e6eca6fd1448647113b3a387802930"}
Oct 30 18:39:58.703733 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:58.703528    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff","Type":"ContainerStarted","Data":"e768772fd63b3fb379509ccbd6c1514d92d26bdb5b3e1eedaad3450073308bbb"}
Oct 30 18:39:59.473585 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:59.473554    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d615363f-bfb2-4a9f-8678-774d3effd7d6" path="/var/lib/kubelet/pods/d615363f-bfb2-4a9f-8678-774d3effd7d6/volumes"
Oct 30 18:39:59.710608 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:59.710573    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff","Type":"ContainerStarted","Data":"e07a70159a7fce817e9af44da61be97ba5b3c61073c9d480055be4e07b9db773"}
Oct 30 18:39:59.710989 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:59.710635    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff","Type":"ContainerStarted","Data":"f648dba484397b16aec6eeabf90e7b3f28a24a11ac4a42b307b4dc8ccee13d53"}
Oct 30 18:39:59.710989 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:59.710663    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff","Type":"ContainerStarted","Data":"a610e9fd77524bf8f8becd5261d17f49f1cd940c239d67f8e0396f0228ee84d4"}
Oct 30 18:39:59.710989 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:59.710686    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff","Type":"ContainerStarted","Data":"ca0a5c9f3b6fb0de31b6664debb57e42a40061bd5b5aa917f77b609d05e981e2"}
Oct 30 18:39:59.710989 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:59.710714    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff","Type":"ContainerStarted","Data":"1c8d6d7d971535b21252fbda946f3b7b4b7e54d04ed4a990a2700015a4ba7372"}
Oct 30 18:39:59.710989 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:59.710738    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff","Type":"ContainerStarted","Data":"8e2b93740b4eb9b2fad1e9230e67a16b60653b9978a56744ad36c903ee9f9248"}
Oct 30 18:39:59.834759 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:39:59.834694    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.818654654 podStartE2EDuration="2.834672487s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.017833ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:59.833442867 +0000 UTC m=+38.476610483" watchObservedRunningTime="2026-10-30 18:39:59.834672487 +0000 UTC m=+38.477840102"
Oct 30 18:40:03.172027 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:40:03.171990    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:40:23.256615 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:40:23.256579    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:40:49.777297 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:40:49.777262    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:40:49.793498 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:40:49.793472    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:40:49.914116 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:40:49.914083    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:40:53.256504 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:40:53.256467    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:41:23.257482 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:41:23.257397    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:41:53.256762 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:41:53.256725    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:42:23.255871 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:42:23.255837    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:42:53.257341 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:42:53.257247    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:42:56.064903 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.064870    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7"]
Oct 30 18:42:56.068219 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.068200    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7"
Oct 30 18:42:56.167598 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.167563    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jblp\" (UniqueName: \"kubernetes.io/projected/28503a01-f236-4d2c-a0ab-722ffad1800f-kube-api-access-4jblp\") pod \"openshift-lws-operator-8465994787-qcdh7\" (UID: \"28503a01-f236-4d2c-a0ab-722ffad1800f\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7"
Oct 30 18:42:56.167805 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.167728    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28503a01-f236-4d2c-a0ab-722ffad1800f-tmp\") pod \"openshift-lws-operator-8465994787-qcdh7\" (UID: \"28503a01-f236-4d2c-a0ab-722ffad1800f\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7"
Oct 30 18:42:56.169793 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.169776    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 18:42:56.169863 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.169791    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-26f2l\""
Oct 30 18:42:56.169863 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.169811    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:42:56.170565 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.170544    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7"]
Oct 30 18:42:56.268700 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.268664    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jblp\" (UniqueName: \"kubernetes.io/projected/28503a01-f236-4d2c-a0ab-722ffad1800f-kube-api-access-4jblp\") pod \"openshift-lws-operator-8465994787-qcdh7\" (UID: \"28503a01-f236-4d2c-a0ab-722ffad1800f\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7"
Oct 30 18:42:56.268888 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.268755    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28503a01-f236-4d2c-a0ab-722ffad1800f-tmp\") pod \"openshift-lws-operator-8465994787-qcdh7\" (UID: \"28503a01-f236-4d2c-a0ab-722ffad1800f\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7"
Oct 30 18:42:56.269131 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.269115    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28503a01-f236-4d2c-a0ab-722ffad1800f-tmp\") pod \"openshift-lws-operator-8465994787-qcdh7\" (UID: \"28503a01-f236-4d2c-a0ab-722ffad1800f\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7"
Oct 30 18:42:56.325976 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.325908    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jblp\" (UniqueName: \"kubernetes.io/projected/28503a01-f236-4d2c-a0ab-722ffad1800f-kube-api-access-4jblp\") pod \"openshift-lws-operator-8465994787-qcdh7\" (UID: \"28503a01-f236-4d2c-a0ab-722ffad1800f\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7"
Oct 30 18:42:56.390773 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.390736    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7"
Oct 30 18:42:56.759452 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.759381    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7"]
Oct 30 18:42:56.762743 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:42:56.762716    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28503a01_f236_4d2c_a0ab_722ffad1800f.slice/crio-0c09e4f6c864c2997af38ad8a7a24132354af25abfe1cc7ead8a7799a4b604a0 WatchSource:0}: Error finding container 0c09e4f6c864c2997af38ad8a7a24132354af25abfe1cc7ead8a7799a4b604a0: Status 404 returned error can't find the container with id 0c09e4f6c864c2997af38ad8a7a24132354af25abfe1cc7ead8a7799a4b604a0
Oct 30 18:42:56.764624 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:56.764604    6014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:42:57.364251 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:42:57.364216    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7" event={"ID":"28503a01-f236-4d2c-a0ab-722ffad1800f","Type":"ContainerStarted","Data":"0c09e4f6c864c2997af38ad8a7a24132354af25abfe1cc7ead8a7799a4b604a0"}
Oct 30 18:43:01.382512 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:01.382466    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7" event={"ID":"28503a01-f236-4d2c-a0ab-722ffad1800f","Type":"ContainerStarted","Data":"bc8403b487126b8ea1078a1fe6648a48dda5d3ef64e0b2714047b32aef6b729a"}
Oct 30 18:43:01.496287 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:01.496220    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-qcdh7" podStartSLOduration=1.42755645 podStartE2EDuration="5.496202203s" totalImagesPullingTime="4.068645753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:01.495207411 +0000 UTC m=+220.138375024" watchObservedRunningTime="2026-10-30 18:43:01.496202203 +0000 UTC m=+220.139369819"
Oct 30 18:43:16.791919 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.791879    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"]
Oct 30 18:43:16.795241 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.795224    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.848100 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.848063    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66cl9\" (UniqueName: \"kubernetes.io/projected/a0083dba-963f-4e40-a5cf-b53d53563918-kube-api-access-66cl9\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.848100 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.848119    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0083dba-963f-4e40-a5cf-b53d53563918-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.848471 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.848239    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a0083dba-963f-4e40-a5cf-b53d53563918-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.848471 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.848265    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a0083dba-963f-4e40-a5cf-b53d53563918-manifests\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.848471 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.848291    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0083dba-963f-4e40-a5cf-b53d53563918-tmp\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.848471 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.848351    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0083dba-963f-4e40-a5cf-b53d53563918-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.949729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.949695    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0083dba-963f-4e40-a5cf-b53d53563918-tmp\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.949950 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.949743    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0083dba-963f-4e40-a5cf-b53d53563918-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.949950 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.949795    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66cl9\" (UniqueName: \"kubernetes.io/projected/a0083dba-963f-4e40-a5cf-b53d53563918-kube-api-access-66cl9\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.950096 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.949976    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0083dba-963f-4e40-a5cf-b53d53563918-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.950096 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.950014    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a0083dba-963f-4e40-a5cf-b53d53563918-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.950096 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.950045    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a0083dba-963f-4e40-a5cf-b53d53563918-manifests\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.950374 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.950348    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 18:43:16.950494 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.950431    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a0083dba-963f-4e40-a5cf-b53d53563918-manifests\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.950651 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.950556    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 18:43:16.950785 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.950760    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 18:43:16.950867 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.950816    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wzclj\""
Oct 30 18:43:16.950912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.950876    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 18:43:16.950912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.950893    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 18:43:16.952336 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.952318    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0083dba-963f-4e40-a5cf-b53d53563918-tmp\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.962578 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.962553    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0083dba-963f-4e40-a5cf-b53d53563918-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.962713 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.962679    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0083dba-963f-4e40-a5cf-b53d53563918-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.962764 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.962714    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a0083dba-963f-4e40-a5cf-b53d53563918-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:16.977291 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:16.977264    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"]
Oct 30 18:43:17.028448 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:17.028416    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66cl9\" (UniqueName: \"kubernetes.io/projected/a0083dba-963f-4e40-a5cf-b53d53563918-kube-api-access-66cl9\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dnd2f\" (UID: \"a0083dba-963f-4e40-a5cf-b53d53563918\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:17.109491 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:17.109456    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:17.650837 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:17.650806    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"]
Oct 30 18:43:17.654301 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:43:17.654267    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0083dba_963f_4e40_a5cf_b53d53563918.slice/crio-ddc4eec8191d6921da45e2bb7771b02e7c92092e2acac1314ab4ba9d8ec86749 WatchSource:0}: Error finding container ddc4eec8191d6921da45e2bb7771b02e7c92092e2acac1314ab4ba9d8ec86749: Status 404 returned error can't find the container with id ddc4eec8191d6921da45e2bb7771b02e7c92092e2acac1314ab4ba9d8ec86749
Oct 30 18:43:18.451274 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:18.451235    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f" event={"ID":"a0083dba-963f-4e40-a5cf-b53d53563918","Type":"ContainerStarted","Data":"ddc4eec8191d6921da45e2bb7771b02e7c92092e2acac1314ab4ba9d8ec86749"}
Oct 30 18:43:21.467805 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:21.467774    6014 generic.go:356] "Generic (PLEG): container finished" podID="a0083dba-963f-4e40-a5cf-b53d53563918" containerID="02fba52f942361c3b4718b86d90eefe583e98575a6e202b5f1d2574315165c97" exitCode=0
Oct 30 18:43:21.468215 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:21.467851    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f" event={"ID":"a0083dba-963f-4e40-a5cf-b53d53563918","Type":"ContainerDied","Data":"02fba52f942361c3b4718b86d90eefe583e98575a6e202b5f1d2574315165c97"}
Oct 30 18:43:22.473801 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:22.473764    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f" event={"ID":"a0083dba-963f-4e40-a5cf-b53d53563918","Type":"ContainerStarted","Data":"10c4b45fdae268f2f2cd566ccc1076c3c9e53b6ca621f9470937e784ca7d60ee"}
Oct 30 18:43:22.474312 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:22.473896    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:22.601850 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:22.601791    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f" podStartSLOduration=2.07147636 podStartE2EDuration="6.601778429s" totalImagesPullingTime="4.118981446s" totalInitContainerRuntime="411.320623ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:22.600107866 +0000 UTC m=+241.243275480" watchObservedRunningTime="2026-10-30 18:43:22.601778429 +0000 UTC m=+241.244946042"
Oct 30 18:43:23.257360 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:43:23.257314    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:43:33.479245 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:33.479210    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dnd2f"
Oct 30 18:43:40.485628 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.485596    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"]
Oct 30 18:43:40.491957 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.491934    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:40.547204 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.547175    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"]
Oct 30 18:43:40.595616 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.595573    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 18:43:40.595790 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.595635    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 18:43:40.595790 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.595671    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 18:43:40.595790 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.595719    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tvjsk\""
Oct 30 18:43:40.661833 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.661797    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbb09ba7-fdb4-481b-acf6-ebc4df7db86d-metrics-cert\") pod \"lws-controller-manager-cf57f8b59-rhdlx\" (UID: \"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:40.662029 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.661841    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbb09ba7-fdb4-481b-acf6-ebc4df7db86d-cert\") pod \"lws-controller-manager-cf57f8b59-rhdlx\" (UID: \"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:40.662029 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.661871    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh2f7\" (UniqueName: \"kubernetes.io/projected/cbb09ba7-fdb4-481b-acf6-ebc4df7db86d-kube-api-access-fh2f7\") pod \"lws-controller-manager-cf57f8b59-rhdlx\" (UID: \"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:40.662029 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.662014    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cbb09ba7-fdb4-481b-acf6-ebc4df7db86d-manager-config\") pod \"lws-controller-manager-cf57f8b59-rhdlx\" (UID: \"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:40.762541 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.762455    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cbb09ba7-fdb4-481b-acf6-ebc4df7db86d-manager-config\") pod \"lws-controller-manager-cf57f8b59-rhdlx\" (UID: \"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:40.762541 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.762512    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbb09ba7-fdb4-481b-acf6-ebc4df7db86d-metrics-cert\") pod \"lws-controller-manager-cf57f8b59-rhdlx\" (UID: \"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:40.762541 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.762532    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbb09ba7-fdb4-481b-acf6-ebc4df7db86d-cert\") pod \"lws-controller-manager-cf57f8b59-rhdlx\" (UID: \"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:40.762760 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.762547    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh2f7\" (UniqueName: \"kubernetes.io/projected/cbb09ba7-fdb4-481b-acf6-ebc4df7db86d-kube-api-access-fh2f7\") pod \"lws-controller-manager-cf57f8b59-rhdlx\" (UID: \"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:40.763261 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.763232    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cbb09ba7-fdb4-481b-acf6-ebc4df7db86d-manager-config\") pod \"lws-controller-manager-cf57f8b59-rhdlx\" (UID: \"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:40.765273 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.765243    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbb09ba7-fdb4-481b-acf6-ebc4df7db86d-cert\") pod \"lws-controller-manager-cf57f8b59-rhdlx\" (UID: \"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:40.765400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.765381    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbb09ba7-fdb4-481b-acf6-ebc4df7db86d-metrics-cert\") pod \"lws-controller-manager-cf57f8b59-rhdlx\" (UID: \"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:40.819079 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:40.819046    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh2f7\" (UniqueName: \"kubernetes.io/projected/cbb09ba7-fdb4-481b-acf6-ebc4df7db86d-kube-api-access-fh2f7\") pod \"lws-controller-manager-cf57f8b59-rhdlx\" (UID: \"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:41.102474 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:41.102430    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:41.480620 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:41.480547    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"]
Oct 30 18:43:41.484528 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:43:41.484501    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb09ba7_fdb4_481b_acf6_ebc4df7db86d.slice/crio-79c3d203a1b7abe97e9380cb69a5597053cef1d74d27a8d9fbaf3270229df66a WatchSource:0}: Error finding container 79c3d203a1b7abe97e9380cb69a5597053cef1d74d27a8d9fbaf3270229df66a: Status 404 returned error can't find the container with id 79c3d203a1b7abe97e9380cb69a5597053cef1d74d27a8d9fbaf3270229df66a
Oct 30 18:43:41.553373 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:41.553337    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx" event={"ID":"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d","Type":"ContainerStarted","Data":"79c3d203a1b7abe97e9380cb69a5597053cef1d74d27a8d9fbaf3270229df66a"}
Oct 30 18:43:45.572737 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:45.572696    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx" event={"ID":"cbb09ba7-fdb4-481b-acf6-ebc4df7db86d","Type":"ContainerStarted","Data":"f98bed1c94cbec60edd0d9176e37739caa7ca4e7bbad8b8abe2ee529bbeda67c"}
Oct 30 18:43:45.573198 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:45.572798    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:45.685598 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:45.685544    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx" podStartSLOduration=2.5317354 podStartE2EDuration="5.685523084s" totalImagesPullingTime="3.153787684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:45.68524766 +0000 UTC m=+264.328415275" watchObservedRunningTime="2026-10-30 18:43:45.685523084 +0000 UTC m=+264.328690698"
Oct 30 18:43:48.341858 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.341821    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"]
Oct 30 18:43:48.344712 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.344688    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.421708 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.421679    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-w2w9n\""
Oct 30 18:43:48.421888 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.421740    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:43:48.421888 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.421740    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:43:48.435347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.435312    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.435537 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.435359    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b518da-36c0-4652-b3bd-e1053a1226a3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.435537 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.435396    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.435537 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.435472    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.435667 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.435596    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b518da-36c0-4652-b3bd-e1053a1226a3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.435667 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.435644    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b518da-36c0-4652-b3bd-e1053a1226a3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.435807 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.435780    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.435858 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.435823    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzk2v\" (UniqueName: \"kubernetes.io/projected/34b518da-36c0-4652-b3bd-e1053a1226a3-kube-api-access-kzk2v\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.435916 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.435888    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.445686 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.445664    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 18:43:48.448595 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.448570    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"]
Oct 30 18:43:48.536935 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.536896    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.536935 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.536941    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.537238 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.536970    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b518da-36c0-4652-b3bd-e1053a1226a3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.537238 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.537011    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b518da-36c0-4652-b3bd-e1053a1226a3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.537238 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.537064    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.537238 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.537087    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzk2v\" (UniqueName: \"kubernetes.io/projected/34b518da-36c0-4652-b3bd-e1053a1226a3-kube-api-access-kzk2v\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.537238 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.537112    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.537238 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.537188    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.537238 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.537218    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b518da-36c0-4652-b3bd-e1053a1226a3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.537587 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.537552    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.537665 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.537638    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.537727 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.537649    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.537727 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.537711    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.537990 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.537870    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b518da-36c0-4652-b3bd-e1053a1226a3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.539483 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.539457    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b518da-36c0-4652-b3bd-e1053a1226a3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.539761 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.539743    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b518da-36c0-4652-b3bd-e1053a1226a3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.593979 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.593897    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b518da-36c0-4652-b3bd-e1053a1226a3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.594217 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.594092    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzk2v\" (UniqueName: \"kubernetes.io/projected/34b518da-36c0-4652-b3bd-e1053a1226a3-kube-api-access-kzk2v\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5zffhh\" (UID: \"34b518da-36c0-4652-b3bd-e1053a1226a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:48.656362 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:48.656330    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:49.076981 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:49.076951    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"]
Oct 30 18:43:49.080519 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:43:49.080482    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b518da_36c0_4652_b3bd_e1053a1226a3.slice/crio-731f4d28b7ece8cf73a8206f7782c048aa911c4be7c3e8587b17c377d92a7e3d WatchSource:0}: Error finding container 731f4d28b7ece8cf73a8206f7782c048aa911c4be7c3e8587b17c377d92a7e3d: Status 404 returned error can't find the container with id 731f4d28b7ece8cf73a8206f7782c048aa911c4be7c3e8587b17c377d92a7e3d
Oct 30 18:43:49.592846 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:49.592805    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh" event={"ID":"34b518da-36c0-4652-b3bd-e1053a1226a3","Type":"ContainerStarted","Data":"731f4d28b7ece8cf73a8206f7782c048aa911c4be7c3e8587b17c377d92a7e3d"}
Oct 30 18:43:50.403164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:50.403104    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-nj87z"]
Oct 30 18:43:50.406024 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:50.406007    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-nj87z"
Oct 30 18:43:50.458013 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:50.457978    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\""
Oct 30 18:43:50.507527 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:50.507479    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-bwzqw\""
Oct 30 18:43:50.508352 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:50.508326    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-nj87z"]
Oct 30 18:43:50.557648 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:50.557607    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bqph\" (UniqueName: \"kubernetes.io/projected/3794243f-5a1c-4529-ab37-87509cd04491-kube-api-access-5bqph\") pod \"odh-observability-584f99c788-nj87z\" (UID: \"3794243f-5a1c-4529-ab37-87509cd04491\") " pod="opendatahub/odh-observability-584f99c788-nj87z"
Oct 30 18:43:50.557841 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:50.557667    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/3794243f-5a1c-4529-ab37-87509cd04491-webhook-certs\") pod \"odh-observability-584f99c788-nj87z\" (UID: \"3794243f-5a1c-4529-ab37-87509cd04491\") " pod="opendatahub/odh-observability-584f99c788-nj87z"
Oct 30 18:43:50.658825 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:50.658731    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bqph\" (UniqueName: \"kubernetes.io/projected/3794243f-5a1c-4529-ab37-87509cd04491-kube-api-access-5bqph\") pod \"odh-observability-584f99c788-nj87z\" (UID: \"3794243f-5a1c-4529-ab37-87509cd04491\") " pod="opendatahub/odh-observability-584f99c788-nj87z"
Oct 30 18:43:50.659249 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:50.658835    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/3794243f-5a1c-4529-ab37-87509cd04491-webhook-certs\") pod \"odh-observability-584f99c788-nj87z\" (UID: \"3794243f-5a1c-4529-ab37-87509cd04491\") " pod="opendatahub/odh-observability-584f99c788-nj87z"
Oct 30 18:43:50.659249 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:43:50.659018    6014 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 18:43:50.659249 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:43:50.659106    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3794243f-5a1c-4529-ab37-87509cd04491-webhook-certs podName:3794243f-5a1c-4529-ab37-87509cd04491 nodeName:}" failed. No retries permitted until 2026-10-30 18:43:51.159083757 +0000 UTC m=+269.802251368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/3794243f-5a1c-4529-ab37-87509cd04491-webhook-certs") pod "odh-observability-584f99c788-nj87z" (UID: "3794243f-5a1c-4529-ab37-87509cd04491") : secret "odh-observability-webhook-cert" not found
Oct 30 18:43:50.716811 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:50.716775    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bqph\" (UniqueName: \"kubernetes.io/projected/3794243f-5a1c-4529-ab37-87509cd04491-kube-api-access-5bqph\") pod \"odh-observability-584f99c788-nj87z\" (UID: \"3794243f-5a1c-4529-ab37-87509cd04491\") " pod="opendatahub/odh-observability-584f99c788-nj87z"
Oct 30 18:43:51.164010 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:51.163962    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/3794243f-5a1c-4529-ab37-87509cd04491-webhook-certs\") pod \"odh-observability-584f99c788-nj87z\" (UID: \"3794243f-5a1c-4529-ab37-87509cd04491\") " pod="opendatahub/odh-observability-584f99c788-nj87z"
Oct 30 18:43:51.164208 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:43:51.164155    6014 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 18:43:51.164276 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:43:51.164237    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3794243f-5a1c-4529-ab37-87509cd04491-webhook-certs podName:3794243f-5a1c-4529-ab37-87509cd04491 nodeName:}" failed. No retries permitted until 2026-10-30 18:43:52.164216287 +0000 UTC m=+270.807383883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/3794243f-5a1c-4529-ab37-87509cd04491-webhook-certs") pod "odh-observability-584f99c788-nj87z" (UID: "3794243f-5a1c-4529-ab37-87509cd04491") : secret "odh-observability-webhook-cert" not found
Oct 30 18:43:52.172650 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:52.172617    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/3794243f-5a1c-4529-ab37-87509cd04491-webhook-certs\") pod \"odh-observability-584f99c788-nj87z\" (UID: \"3794243f-5a1c-4529-ab37-87509cd04491\") " pod="opendatahub/odh-observability-584f99c788-nj87z"
Oct 30 18:43:52.175636 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:52.175610    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/3794243f-5a1c-4529-ab37-87509cd04491-webhook-certs\") pod \"odh-observability-584f99c788-nj87z\" (UID: \"3794243f-5a1c-4529-ab37-87509cd04491\") " pod="opendatahub/odh-observability-584f99c788-nj87z"
Oct 30 18:43:52.218570 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:52.218537    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-nj87z"
Oct 30 18:43:52.597238 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:52.597202    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-nj87z"]
Oct 30 18:43:52.608222 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:52.608187    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-nj87z" event={"ID":"3794243f-5a1c-4529-ab37-87509cd04491","Type":"ContainerStarted","Data":"20e82b342dd5868badff521b3a82cd24e72ec9a4ede94b1dd0f2bbb4045042d4"}
Oct 30 18:43:52.818699 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:52.818660    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:52.818810 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:52.818733    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:52.818810 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:52.818785    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:53.257063 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:43:53.256975    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:43:53.613500 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:53.613464    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh" event={"ID":"34b518da-36c0-4652-b3bd-e1053a1226a3","Type":"ContainerStarted","Data":"342e30453711f061bc51f8443249922b66c797bbbba7420ade84a49219fad3bb"}
Oct 30 18:43:53.657124 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:53.657093    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:53.661848 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:53.661815    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:53.827990 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:53.827937    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh" podStartSLOduration=2.092859696 podStartE2EDuration="5.827921371s" totalImagesPullingTime="3.735061675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:53.778171144 +0000 UTC m=+272.421338757" watchObservedRunningTime="2026-10-30 18:43:53.827921371 +0000 UTC m=+272.471088984"
Oct 30 18:43:54.619739 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:54.619705    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:54.620697 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:54.620665    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5zffhh"
Oct 30 18:43:56.579833 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:56.579796    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-rhdlx"
Oct 30 18:43:56.629209 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:56.629115    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-nj87z" event={"ID":"3794243f-5a1c-4529-ab37-87509cd04491","Type":"ContainerStarted","Data":"bb80616eed294d59475505e3e8285b0da8b76d58f7099a0e01671c5820c197c8"}
Oct 30 18:43:56.629400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:56.629384    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-nj87z"
Oct 30 18:43:56.818448 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:43:56.818386    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-nj87z" podStartSLOduration=3.065413509 podStartE2EDuration="6.818370986s" totalImagesPullingTime="3.752957477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:56.81813026 +0000 UTC m=+275.461297874" watchObservedRunningTime="2026-10-30 18:43:56.818370986 +0000 UTC m=+275.461538603"
Oct 30 18:44:07.057101 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.057018    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"]
Oct 30 18:44:07.061698 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.061677    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.110648 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.110615    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6sdx\" (UniqueName: \"kubernetes.io/projected/f3c033e3-63bc-4373-a7bf-545fbe3a7f40-kube-api-access-h6sdx\") pod \"kserve-module-controller-manager-5cfb77bf8c-cc5rr\" (UID: \"f3c033e3-63bc-4373-a7bf-545fbe3a7f40\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.110850 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.110679    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f3c033e3-63bc-4373-a7bf-545fbe3a7f40-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-cc5rr\" (UID: \"f3c033e3-63bc-4373-a7bf-545fbe3a7f40\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.110850 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.110710    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3c033e3-63bc-4373-a7bf-545fbe3a7f40-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-cc5rr\" (UID: \"f3c033e3-63bc-4373-a7bf-545fbe3a7f40\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.110850 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.110814    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3c033e3-63bc-4373-a7bf-545fbe3a7f40-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-cc5rr\" (UID: \"f3c033e3-63bc-4373-a7bf-545fbe3a7f40\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.211921 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.211888    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6sdx\" (UniqueName: \"kubernetes.io/projected/f3c033e3-63bc-4373-a7bf-545fbe3a7f40-kube-api-access-h6sdx\") pod \"kserve-module-controller-manager-5cfb77bf8c-cc5rr\" (UID: \"f3c033e3-63bc-4373-a7bf-545fbe3a7f40\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.211921 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.211929    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f3c033e3-63bc-4373-a7bf-545fbe3a7f40-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-cc5rr\" (UID: \"f3c033e3-63bc-4373-a7bf-545fbe3a7f40\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.212286 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.211945    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3c033e3-63bc-4373-a7bf-545fbe3a7f40-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-cc5rr\" (UID: \"f3c033e3-63bc-4373-a7bf-545fbe3a7f40\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.212286 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.212049    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3c033e3-63bc-4373-a7bf-545fbe3a7f40-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-cc5rr\" (UID: \"f3c033e3-63bc-4373-a7bf-545fbe3a7f40\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.212603 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.212579    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f3c033e3-63bc-4373-a7bf-545fbe3a7f40-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-cc5rr\" (UID: \"f3c033e3-63bc-4373-a7bf-545fbe3a7f40\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.212718 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.212608    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3c033e3-63bc-4373-a7bf-545fbe3a7f40-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-cc5rr\" (UID: \"f3c033e3-63bc-4373-a7bf-545fbe3a7f40\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.262862 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.262828    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 18:44:07.263074 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.263055    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-4nq9t\""
Oct 30 18:44:07.264058 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.264033    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"]
Oct 30 18:44:07.268693 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.268670    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6sdx\" (UniqueName: \"kubernetes.io/projected/f3c033e3-63bc-4373-a7bf-545fbe3a7f40-kube-api-access-h6sdx\") pod \"kserve-module-controller-manager-5cfb77bf8c-cc5rr\" (UID: \"f3c033e3-63bc-4373-a7bf-545fbe3a7f40\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.273496 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.273471    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3c033e3-63bc-4373-a7bf-545fbe3a7f40-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-cc5rr\" (UID: \"f3c033e3-63bc-4373-a7bf-545fbe3a7f40\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.373997 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.373962    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:07.636095 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.636017    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-nj87z"
Oct 30 18:44:07.794131 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:07.794095    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"]
Oct 30 18:44:07.802021 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:44:07.801993    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c033e3_63bc_4373_a7bf_545fbe3a7f40.slice/crio-e1ea73177940a9f26461f72e5f61ff363edb4bdae30e20f28f042d337c30b754 WatchSource:0}: Error finding container e1ea73177940a9f26461f72e5f61ff363edb4bdae30e20f28f042d337c30b754: Status 404 returned error can't find the container with id e1ea73177940a9f26461f72e5f61ff363edb4bdae30e20f28f042d337c30b754
Oct 30 18:44:08.684355 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:08.684303    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr" event={"ID":"f3c033e3-63bc-4373-a7bf-545fbe3a7f40","Type":"ContainerStarted","Data":"e1ea73177940a9f26461f72e5f61ff363edb4bdae30e20f28f042d337c30b754"}
Oct 30 18:44:10.695164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:10.695098    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr" event={"ID":"f3c033e3-63bc-4373-a7bf-545fbe3a7f40","Type":"ContainerStarted","Data":"92d374a39ffc92781a822e5d9c149a0ff723843a57d5684dc4b52cfd19d2d3d7"}
Oct 30 18:44:10.695645 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:10.695269    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:10.859787 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:10.859731    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr" podStartSLOduration=1.131667932 podStartE2EDuration="3.859711719s" totalImagesPullingTime="2.728043787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:10.859432694 +0000 UTC m=+289.502600310" watchObservedRunningTime="2026-10-30 18:44:10.859711719 +0000 UTC m=+289.502879333"
Oct 30 18:44:23.256908 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:44:23.256867    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:44:31.700561 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:44:31.700520    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cc5rr"
Oct 30 18:44:53.259169 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:44:53.259108    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:45:04.914958 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:04.914919    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-s42cx"]
Oct 30 18:45:04.918710 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:04.918687    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-s42cx"
Oct 30 18:45:05.015299 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.015244    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd39a056-ee8f-40c4-8837-bea0bbd41085-cert\") pod \"odh-model-controller-f87fcf56c-s42cx\" (UID: \"bd39a056-ee8f-40c4-8837-bea0bbd41085\") " pod="opendatahub/odh-model-controller-f87fcf56c-s42cx"
Oct 30 18:45:05.015299 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.015294    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw2bc\" (UniqueName: \"kubernetes.io/projected/bd39a056-ee8f-40c4-8837-bea0bbd41085-kube-api-access-hw2bc\") pod \"odh-model-controller-f87fcf56c-s42cx\" (UID: \"bd39a056-ee8f-40c4-8837-bea0bbd41085\") " pod="opendatahub/odh-model-controller-f87fcf56c-s42cx"
Oct 30 18:45:05.015539 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.015443    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39a056-ee8f-40c4-8837-bea0bbd41085-metrics-certs\") pod \"odh-model-controller-f87fcf56c-s42cx\" (UID: \"bd39a056-ee8f-40c4-8837-bea0bbd41085\") " pod="opendatahub/odh-model-controller-f87fcf56c-s42cx"
Oct 30 18:45:05.020167 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.020121    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 18:45:05.020167 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.020121    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-drb7w\""
Oct 30 18:45:05.020318 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.020172    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 18:45:05.021097 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.021081    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-s42cx"]
Oct 30 18:45:05.116107 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.116062    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39a056-ee8f-40c4-8837-bea0bbd41085-metrics-certs\") pod \"odh-model-controller-f87fcf56c-s42cx\" (UID: \"bd39a056-ee8f-40c4-8837-bea0bbd41085\") " pod="opendatahub/odh-model-controller-f87fcf56c-s42cx"
Oct 30 18:45:05.116107 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.116117    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd39a056-ee8f-40c4-8837-bea0bbd41085-cert\") pod \"odh-model-controller-f87fcf56c-s42cx\" (UID: \"bd39a056-ee8f-40c4-8837-bea0bbd41085\") " pod="opendatahub/odh-model-controller-f87fcf56c-s42cx"
Oct 30 18:45:05.116394 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.116257    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw2bc\" (UniqueName: \"kubernetes.io/projected/bd39a056-ee8f-40c4-8837-bea0bbd41085-kube-api-access-hw2bc\") pod \"odh-model-controller-f87fcf56c-s42cx\" (UID: \"bd39a056-ee8f-40c4-8837-bea0bbd41085\") " pod="opendatahub/odh-model-controller-f87fcf56c-s42cx"
Oct 30 18:45:05.118732 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.118708    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39a056-ee8f-40c4-8837-bea0bbd41085-metrics-certs\") pod \"odh-model-controller-f87fcf56c-s42cx\" (UID: \"bd39a056-ee8f-40c4-8837-bea0bbd41085\") " pod="opendatahub/odh-model-controller-f87fcf56c-s42cx"
Oct 30 18:45:05.118732 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.118726    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd39a056-ee8f-40c4-8837-bea0bbd41085-cert\") pod \"odh-model-controller-f87fcf56c-s42cx\" (UID: \"bd39a056-ee8f-40c4-8837-bea0bbd41085\") " pod="opendatahub/odh-model-controller-f87fcf56c-s42cx"
Oct 30 18:45:05.173036 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.172964    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw2bc\" (UniqueName: \"kubernetes.io/projected/bd39a056-ee8f-40c4-8837-bea0bbd41085-kube-api-access-hw2bc\") pod \"odh-model-controller-f87fcf56c-s42cx\" (UID: \"bd39a056-ee8f-40c4-8837-bea0bbd41085\") " pod="opendatahub/odh-model-controller-f87fcf56c-s42cx"
Oct 30 18:45:05.231160 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.231110    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-s42cx"
Oct 30 18:45:05.598705 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.598674    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-s42cx"]
Oct 30 18:45:05.601752 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:45:05.601729    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd39a056_ee8f_40c4_8837_bea0bbd41085.slice/crio-55e83b6b31f1607e06205a383bfc40ce9d8ddff4b6f38c166d9c2fae6e5f6542 WatchSource:0}: Error finding container 55e83b6b31f1607e06205a383bfc40ce9d8ddff4b6f38c166d9c2fae6e5f6542: Status 404 returned error can't find the container with id 55e83b6b31f1607e06205a383bfc40ce9d8ddff4b6f38c166d9c2fae6e5f6542
Oct 30 18:45:05.939185 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:05.939083    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-s42cx" event={"ID":"bd39a056-ee8f-40c4-8837-bea0bbd41085","Type":"ContainerStarted","Data":"55e83b6b31f1607e06205a383bfc40ce9d8ddff4b6f38c166d9c2fae6e5f6542"}
Oct 30 18:45:08.954869 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:08.954779    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-s42cx" event={"ID":"bd39a056-ee8f-40c4-8837-bea0bbd41085","Type":"ContainerStarted","Data":"7ef5fe8fdc9b4c6357fce56560014ac36142f21f84c3076fff6c6fb443d6c903"}
Oct 30 18:45:08.955371 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:08.954908    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-s42cx"
Oct 30 18:45:09.069308 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:09.069253    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-s42cx" podStartSLOduration=2.103834678 podStartE2EDuration="5.069238519s" totalImagesPullingTime="2.965403841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:09.068746931 +0000 UTC m=+347.711914545" watchObservedRunningTime="2026-10-30 18:45:09.069238519 +0000 UTC m=+347.712406184"
Oct 30 18:45:19.961423 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:45:19.961387    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-s42cx"
Oct 30 18:45:23.257561 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:45:23.257521    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:45:53.256934 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:45:53.256850    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:46:23.257637 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:46:23.257595    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:46:34.097670 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.097634    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"]
Oct 30 18:46:34.101024 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.101006    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:34.134298 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.134260    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxjsg\" (UniqueName: \"kubernetes.io/projected/07c06952-4fdf-408b-9fa7-8044f1dda6e3-kube-api-access-wxjsg\") pod \"kuadrant-operator-controller-manager-6b6fb56846-hblw4\" (UID: \"07c06952-4fdf-408b-9fa7-8044f1dda6e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:34.134454 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.134390    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c06952-4fdf-408b-9fa7-8044f1dda6e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-hblw4\" (UID: \"07c06952-4fdf-408b-9fa7-8044f1dda6e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:34.153545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.153512    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 18:46:34.202819 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.202793    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 18:46:34.203206 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.203185    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l677f\""
Oct 30 18:46:34.204245 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.204226    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"]
Oct 30 18:46:34.234869 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.234834    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c06952-4fdf-408b-9fa7-8044f1dda6e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-hblw4\" (UID: \"07c06952-4fdf-408b-9fa7-8044f1dda6e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:34.235039 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.234905    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxjsg\" (UniqueName: \"kubernetes.io/projected/07c06952-4fdf-408b-9fa7-8044f1dda6e3-kube-api-access-wxjsg\") pod \"kuadrant-operator-controller-manager-6b6fb56846-hblw4\" (UID: \"07c06952-4fdf-408b-9fa7-8044f1dda6e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:34.235343 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.235325    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c06952-4fdf-408b-9fa7-8044f1dda6e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-hblw4\" (UID: \"07c06952-4fdf-408b-9fa7-8044f1dda6e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:34.298016 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.297981    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxjsg\" (UniqueName: \"kubernetes.io/projected/07c06952-4fdf-408b-9fa7-8044f1dda6e3-kube-api-access-wxjsg\") pod \"kuadrant-operator-controller-manager-6b6fb56846-hblw4\" (UID: \"07c06952-4fdf-408b-9fa7-8044f1dda6e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:34.420162 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.420037    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:34.899697 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:34.899655    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"]
Oct 30 18:46:34.953341 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:46:34.953307    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c06952_4fdf_408b_9fa7_8044f1dda6e3.slice/crio-d1753812f1b7c6056a0044442a36d3845ca6687c0632f74e67cab0e260f916d3 WatchSource:0}: Error finding container d1753812f1b7c6056a0044442a36d3845ca6687c0632f74e67cab0e260f916d3: Status 404 returned error can't find the container with id d1753812f1b7c6056a0044442a36d3845ca6687c0632f74e67cab0e260f916d3
Oct 30 18:46:35.325768 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:35.325732    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4" event={"ID":"07c06952-4fdf-408b-9fa7-8044f1dda6e3","Type":"ContainerStarted","Data":"d1753812f1b7c6056a0044442a36d3845ca6687c0632f74e67cab0e260f916d3"}
Oct 30 18:46:38.692357 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:38.692318    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-9qpl9"]
Oct 30 18:46:38.695946 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:38.695922    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-9qpl9"
Oct 30 18:46:38.750010 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:38.749982    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zrvrq\""
Oct 30 18:46:38.778007 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:38.777971    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv4z5\" (UniqueName: \"kubernetes.io/projected/7a1d7873-6426-4c65-bc71-59f5a7ab95cb-kube-api-access-sv4z5\") pod \"authorino-operator-68d6dd76dc-9qpl9\" (UID: \"7a1d7873-6426-4c65-bc71-59f5a7ab95cb\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-9qpl9"
Oct 30 18:46:38.800400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:38.800358    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-9qpl9"]
Oct 30 18:46:38.878788 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:38.878749    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv4z5\" (UniqueName: \"kubernetes.io/projected/7a1d7873-6426-4c65-bc71-59f5a7ab95cb-kube-api-access-sv4z5\") pod \"authorino-operator-68d6dd76dc-9qpl9\" (UID: \"7a1d7873-6426-4c65-bc71-59f5a7ab95cb\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-9qpl9"
Oct 30 18:46:38.939070 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:38.939036    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv4z5\" (UniqueName: \"kubernetes.io/projected/7a1d7873-6426-4c65-bc71-59f5a7ab95cb-kube-api-access-sv4z5\") pod \"authorino-operator-68d6dd76dc-9qpl9\" (UID: \"7a1d7873-6426-4c65-bc71-59f5a7ab95cb\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-9qpl9"
Oct 30 18:46:39.014719 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:39.014629    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-9qpl9"
Oct 30 18:46:39.398091 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:39.398056    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-9qpl9"]
Oct 30 18:46:39.402021 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:46:39.401996    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1d7873_6426_4c65_bc71_59f5a7ab95cb.slice/crio-176b51d3e74d25e5e3e59eeb7e427200eff0b56d6e14da47189ff69feda6337d WatchSource:0}: Error finding container 176b51d3e74d25e5e3e59eeb7e427200eff0b56d6e14da47189ff69feda6337d: Status 404 returned error can't find the container with id 176b51d3e74d25e5e3e59eeb7e427200eff0b56d6e14da47189ff69feda6337d
Oct 30 18:46:40.353701 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:40.353662    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-9qpl9" event={"ID":"7a1d7873-6426-4c65-bc71-59f5a7ab95cb","Type":"ContainerStarted","Data":"176b51d3e74d25e5e3e59eeb7e427200eff0b56d6e14da47189ff69feda6337d"}
Oct 30 18:46:40.355697 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:40.355655    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4" event={"ID":"07c06952-4fdf-408b-9fa7-8044f1dda6e3","Type":"ContainerStarted","Data":"5a694ee3cd68ce9b6889ea9b70f29e0c156128d0b684ab0be636dea9d8ea1f8a"}
Oct 30 18:46:40.355857 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:40.355784    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:40.584498 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:40.584444    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4" podStartSLOduration=1.409043189 podStartE2EDuration="6.584428213s" totalImagesPullingTime="5.175385024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:40.534741221 +0000 UTC m=+439.177908835" watchObservedRunningTime="2026-10-30 18:46:40.584428213 +0000 UTC m=+439.227595826"
Oct 30 18:46:42.368855 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:42.368819    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-9qpl9" event={"ID":"7a1d7873-6426-4c65-bc71-59f5a7ab95cb","Type":"ContainerStarted","Data":"4ee1be24f4a94a08d1b111d76033b87437df21cb24732ae16e3b77485d5ebfec"}
Oct 30 18:46:42.369287 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:42.368896    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-9qpl9"
Oct 30 18:46:42.482424 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:42.482355    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-9qpl9" podStartSLOduration=1.9845937070000002 podStartE2EDuration="4.482339488s" totalImagesPullingTime="2.497745781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:42.481737606 +0000 UTC m=+441.124905221" watchObservedRunningTime="2026-10-30 18:46:42.482339488 +0000 UTC m=+441.125507103"
Oct 30 18:46:51.364291 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:51.364259    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:53.258223 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:46:53.258186    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:46:53.376850 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:53.376819    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-9qpl9"
Oct 30 18:46:55.138566 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.138520    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"]
Oct 30 18:46:55.138954 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.138793    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4" podUID="07c06952-4fdf-408b-9fa7-8044f1dda6e3" containerName="manager" containerID="cri-o://5a694ee3cd68ce9b6889ea9b70f29e0c156128d0b684ab0be636dea9d8ea1f8a" gracePeriod=2
Oct 30 18:46:55.140053 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.140029    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"]
Oct 30 18:46:55.187995 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.187936    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"]
Oct 30 18:46:55.188570 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.188550    6014 cpu_manager.go:414] "containerMap: removing container" podUID="07c06952-4fdf-408b-9fa7-8044f1dda6e3" containerName="manager"
Oct 30 18:46:55.188570 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.188567    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c06952-4fdf-408b-9fa7-8044f1dda6e3" containerName="manager"
Oct 30 18:46:55.188763 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.188653    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c06952-4fdf-408b-9fa7-8044f1dda6e3" containerName="manager"
Oct 30 18:46:55.189344 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:46:55.189308    6014 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-hblw4\" is forbidden: User \"system:node:ip-10-0-143-97.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-97.us-west-1.compute.internal' and this object" podUID="07c06952-4fdf-408b-9fa7-8044f1dda6e3" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:55.191680 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.191661    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"]
Oct 30 18:46:55.191817 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.191798    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"
Oct 30 18:46:55.194837 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.194820    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"
Oct 30 18:46:55.214572 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.214528    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"]
Oct 30 18:46:55.244810 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.244780    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a73748f-c79c-4b86-b0be-33b914048cc2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-gvz5j\" (UID: \"0a73748f-c79c-4b86-b0be-33b914048cc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"
Oct 30 18:46:55.244952 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.244910    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsfch\" (UniqueName: \"kubernetes.io/projected/0a73748f-c79c-4b86-b0be-33b914048cc2-kube-api-access-dsfch\") pod \"kuadrant-operator-controller-manager-6b6fb56846-gvz5j\" (UID: \"0a73748f-c79c-4b86-b0be-33b914048cc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"
Oct 30 18:46:55.283200 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.281641    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"]
Oct 30 18:46:55.345718 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.345682    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a73748f-c79c-4b86-b0be-33b914048cc2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-gvz5j\" (UID: \"0a73748f-c79c-4b86-b0be-33b914048cc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"
Oct 30 18:46:55.345917 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.345749    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d4e71c5-5b67-4098-af2e-d5218b15ba47-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-bfglz\" (UID: \"7d4e71c5-5b67-4098-af2e-d5218b15ba47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"
Oct 30 18:46:55.345917 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.345779    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-266np\" (UniqueName: \"kubernetes.io/projected/7d4e71c5-5b67-4098-af2e-d5218b15ba47-kube-api-access-266np\") pod \"kuadrant-operator-controller-manager-956956bc4-bfglz\" (UID: \"7d4e71c5-5b67-4098-af2e-d5218b15ba47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"
Oct 30 18:46:55.346039 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.345947    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsfch\" (UniqueName: \"kubernetes.io/projected/0a73748f-c79c-4b86-b0be-33b914048cc2-kube-api-access-dsfch\") pod \"kuadrant-operator-controller-manager-6b6fb56846-gvz5j\" (UID: \"0a73748f-c79c-4b86-b0be-33b914048cc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"
Oct 30 18:46:55.346078 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.346061    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a73748f-c79c-4b86-b0be-33b914048cc2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-gvz5j\" (UID: \"0a73748f-c79c-4b86-b0be-33b914048cc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"
Oct 30 18:46:55.358486 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:46:55.358433    6014 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-hblw4\" is forbidden: User \"system:node:ip-10-0-143-97.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-97.us-west-1.compute.internal' and this object" podUID="07c06952-4fdf-408b-9fa7-8044f1dda6e3" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:55.407383 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.407360    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsfch\" (UniqueName: \"kubernetes.io/projected/0a73748f-c79c-4b86-b0be-33b914048cc2-kube-api-access-dsfch\") pod \"kuadrant-operator-controller-manager-6b6fb56846-gvz5j\" (UID: \"0a73748f-c79c-4b86-b0be-33b914048cc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"
Oct 30 18:46:55.408963 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:46:55.408939    6014 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-hblw4\" is forbidden: User \"system:node:ip-10-0-143-97.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-97.us-west-1.compute.internal' and this object" podUID="07c06952-4fdf-408b-9fa7-8044f1dda6e3" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:55.428028 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.428003    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:55.429567 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.429539    6014 generic.go:356] "Generic (PLEG): container finished" podID="07c06952-4fdf-408b-9fa7-8044f1dda6e3" containerID="5a694ee3cd68ce9b6889ea9b70f29e0c156128d0b684ab0be636dea9d8ea1f8a" exitCode=0
Oct 30 18:46:55.429660 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.429581    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-hblw4"
Oct 30 18:46:55.429660 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.429620    6014 scope.go:122] "RemoveContainer" containerID="5a694ee3cd68ce9b6889ea9b70f29e0c156128d0b684ab0be636dea9d8ea1f8a"
Oct 30 18:46:55.437917 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.437896    6014 scope.go:122] "RemoveContainer" containerID="5a694ee3cd68ce9b6889ea9b70f29e0c156128d0b684ab0be636dea9d8ea1f8a"
Oct 30 18:46:55.438233 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:46:55.438209    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a694ee3cd68ce9b6889ea9b70f29e0c156128d0b684ab0be636dea9d8ea1f8a\": container with ID starting with 5a694ee3cd68ce9b6889ea9b70f29e0c156128d0b684ab0be636dea9d8ea1f8a not found: ID does not exist" containerID="5a694ee3cd68ce9b6889ea9b70f29e0c156128d0b684ab0be636dea9d8ea1f8a"
Oct 30 18:46:55.438292 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.438249    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a694ee3cd68ce9b6889ea9b70f29e0c156128d0b684ab0be636dea9d8ea1f8a"} err="failed to get container status \"5a694ee3cd68ce9b6889ea9b70f29e0c156128d0b684ab0be636dea9d8ea1f8a\": rpc error: code = NotFound desc = could not find container \"5a694ee3cd68ce9b6889ea9b70f29e0c156128d0b684ab0be636dea9d8ea1f8a\": container with ID starting with 5a694ee3cd68ce9b6889ea9b70f29e0c156128d0b684ab0be636dea9d8ea1f8a not found: ID does not exist"
Oct 30 18:46:55.446668 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.446642    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d4e71c5-5b67-4098-af2e-d5218b15ba47-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-bfglz\" (UID: \"7d4e71c5-5b67-4098-af2e-d5218b15ba47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"
Oct 30 18:46:55.446776 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.446681    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266np\" (UniqueName: \"kubernetes.io/projected/7d4e71c5-5b67-4098-af2e-d5218b15ba47-kube-api-access-266np\") pod \"kuadrant-operator-controller-manager-956956bc4-bfglz\" (UID: \"7d4e71c5-5b67-4098-af2e-d5218b15ba47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"
Oct 30 18:46:55.446953 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.446937    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d4e71c5-5b67-4098-af2e-d5218b15ba47-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-bfglz\" (UID: \"7d4e71c5-5b67-4098-af2e-d5218b15ba47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"
Oct 30 18:46:55.504501 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.504472    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-266np\" (UniqueName: \"kubernetes.io/projected/7d4e71c5-5b67-4098-af2e-d5218b15ba47-kube-api-access-266np\") pod \"kuadrant-operator-controller-manager-956956bc4-bfglz\" (UID: \"7d4e71c5-5b67-4098-af2e-d5218b15ba47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"
Oct 30 18:46:55.534688 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.534657    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"
Oct 30 18:46:55.540872 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.540842    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"
Oct 30 18:46:55.548077 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.548049    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c06952-4fdf-408b-9fa7-8044f1dda6e3-kube-api-access-wxjsg\" (UniqueName: \"kubernetes.io/projected/07c06952-4fdf-408b-9fa7-8044f1dda6e3-kube-api-access-wxjsg\") pod \"07c06952-4fdf-408b-9fa7-8044f1dda6e3\" (UID: \"07c06952-4fdf-408b-9fa7-8044f1dda6e3\") "
Oct 30 18:46:55.548214 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.548124    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c06952-4fdf-408b-9fa7-8044f1dda6e3-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c06952-4fdf-408b-9fa7-8044f1dda6e3-extensions-socket-volume\") pod \"07c06952-4fdf-408b-9fa7-8044f1dda6e3\" (UID: \"07c06952-4fdf-408b-9fa7-8044f1dda6e3\") "
Oct 30 18:46:55.548734 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.548708    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c06952-4fdf-408b-9fa7-8044f1dda6e3-extensions-socket-volume" pod "07c06952-4fdf-408b-9fa7-8044f1dda6e3" (UID: "07c06952-4fdf-408b-9fa7-8044f1dda6e3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:46:55.550797 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.550765    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c06952-4fdf-408b-9fa7-8044f1dda6e3-kube-api-access-wxjsg" pod "07c06952-4fdf-408b-9fa7-8044f1dda6e3" (UID: "07c06952-4fdf-408b-9fa7-8044f1dda6e3"). InnerVolumeSpecName "kube-api-access-wxjsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:55.649188 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.649133    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxjsg\" (UniqueName: \"kubernetes.io/projected/07c06952-4fdf-408b-9fa7-8044f1dda6e3-kube-api-access-wxjsg\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:55.649188 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:55.649186    6014 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c06952-4fdf-408b-9fa7-8044f1dda6e3-extensions-socket-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:56.020053 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:56.019969    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"]
Oct 30 18:46:56.020965 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:56.020931    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"]
Oct 30 18:46:56.023468 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:46:56.023436    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a73748f_c79c_4b86_b0be_33b914048cc2.slice/crio-a922dc269160a49256f59a9d9de07863531c867fa4ecabad94ddea28cdf1a848 WatchSource:0}: Error finding container a922dc269160a49256f59a9d9de07863531c867fa4ecabad94ddea28cdf1a848: Status 404 returned error can't find the container with id a922dc269160a49256f59a9d9de07863531c867fa4ecabad94ddea28cdf1a848
Oct 30 18:46:56.024102 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:46:56.024074    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4e71c5_5b67_4098_af2e_d5218b15ba47.slice/crio-9d02e7c1e74b43eee0c65f872860c29b93623d193e18ae56a8d976b88eb53969 WatchSource:0}: Error finding container 9d02e7c1e74b43eee0c65f872860c29b93623d193e18ae56a8d976b88eb53969: Status 404 returned error can't find the container with id 9d02e7c1e74b43eee0c65f872860c29b93623d193e18ae56a8d976b88eb53969
Oct 30 18:46:56.437471 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:56.437434    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz" event={"ID":"7d4e71c5-5b67-4098-af2e-d5218b15ba47","Type":"ContainerStarted","Data":"75fa067235a1c85d7bb3e44839c4bd3e0fb7de42e558183501227194648f3b12"}
Oct 30 18:46:56.437935 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:56.437498    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz" event={"ID":"7d4e71c5-5b67-4098-af2e-d5218b15ba47","Type":"ContainerStarted","Data":"9d02e7c1e74b43eee0c65f872860c29b93623d193e18ae56a8d976b88eb53969"}
Oct 30 18:46:56.437935 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:56.437529    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"
Oct 30 18:46:56.439123 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:56.439101    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j" event={"ID":"0a73748f-c79c-4b86-b0be-33b914048cc2","Type":"ContainerStarted","Data":"33851451ba4fb97c51dd421327866fe269d8b6fbfec59d1d25bec8cc0a37089e"}
Oct 30 18:46:56.439254 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:56.439155    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j" event={"ID":"0a73748f-c79c-4b86-b0be-33b914048cc2","Type":"ContainerStarted","Data":"a922dc269160a49256f59a9d9de07863531c867fa4ecabad94ddea28cdf1a848"}
Oct 30 18:46:56.439254 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:56.439194    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"
Oct 30 18:46:56.683771 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:56.683593    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j" podStartSLOduration=1.683576795 podStartE2EDuration="1.683576795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:56.683346797 +0000 UTC m=+455.326514411" watchObservedRunningTime="2026-10-30 18:46:56.683576795 +0000 UTC m=+455.326744408"
Oct 30 18:46:56.684095 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:56.684070    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz" podStartSLOduration=1.684060461 podStartE2EDuration="1.684060461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:56.559041402 +0000 UTC m=+455.202209031" watchObservedRunningTime="2026-10-30 18:46:56.684060461 +0000 UTC m=+455.327228075"
Oct 30 18:46:57.473923 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:46:57.473890    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c06952-4fdf-408b-9fa7-8044f1dda6e3" path="/var/lib/kubelet/pods/07c06952-4fdf-408b-9fa7-8044f1dda6e3/volumes"
Oct 30 18:47:07.446015 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:07.445944    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"
Oct 30 18:47:07.446497 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:07.446125    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"
Oct 30 18:47:07.770295 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:07.770184    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"]
Oct 30 18:47:07.770584 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:07.770523    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j" podUID="0a73748f-c79c-4b86-b0be-33b914048cc2" containerName="manager" containerID="cri-o://33851451ba4fb97c51dd421327866fe269d8b6fbfec59d1d25bec8cc0a37089e" gracePeriod=10
Oct 30 18:47:08.023271 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.023206    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"
Oct 30 18:47:08.178153 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.178104    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a73748f-c79c-4b86-b0be-33b914048cc2-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a73748f-c79c-4b86-b0be-33b914048cc2-extensions-socket-volume\") pod \"0a73748f-c79c-4b86-b0be-33b914048cc2\" (UID: \"0a73748f-c79c-4b86-b0be-33b914048cc2\") "
Oct 30 18:47:08.178329 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.178187    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a73748f-c79c-4b86-b0be-33b914048cc2-kube-api-access-dsfch\" (UniqueName: \"kubernetes.io/projected/0a73748f-c79c-4b86-b0be-33b914048cc2-kube-api-access-dsfch\") pod \"0a73748f-c79c-4b86-b0be-33b914048cc2\" (UID: \"0a73748f-c79c-4b86-b0be-33b914048cc2\") "
Oct 30 18:47:08.178567 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.178538    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a73748f-c79c-4b86-b0be-33b914048cc2-extensions-socket-volume" pod "0a73748f-c79c-4b86-b0be-33b914048cc2" (UID: "0a73748f-c79c-4b86-b0be-33b914048cc2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:47:08.180318 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.180289    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a73748f-c79c-4b86-b0be-33b914048cc2-kube-api-access-dsfch" pod "0a73748f-c79c-4b86-b0be-33b914048cc2" (UID: "0a73748f-c79c-4b86-b0be-33b914048cc2"). InnerVolumeSpecName "kube-api-access-dsfch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:08.279870 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.279772    6014 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a73748f-c79c-4b86-b0be-33b914048cc2-extensions-socket-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:47:08.279870 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.279805    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsfch\" (UniqueName: \"kubernetes.io/projected/0a73748f-c79c-4b86-b0be-33b914048cc2-kube-api-access-dsfch\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:47:08.497487 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.497451    6014 generic.go:356] "Generic (PLEG): container finished" podID="0a73748f-c79c-4b86-b0be-33b914048cc2" containerID="33851451ba4fb97c51dd421327866fe269d8b6fbfec59d1d25bec8cc0a37089e" exitCode=0
Oct 30 18:47:08.497909 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.497504    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"
Oct 30 18:47:08.497909 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.497534    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j" event={"ID":"0a73748f-c79c-4b86-b0be-33b914048cc2","Type":"ContainerDied","Data":"33851451ba4fb97c51dd421327866fe269d8b6fbfec59d1d25bec8cc0a37089e"}
Oct 30 18:47:08.497909 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.497590    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j" event={"ID":"0a73748f-c79c-4b86-b0be-33b914048cc2","Type":"ContainerDied","Data":"a922dc269160a49256f59a9d9de07863531c867fa4ecabad94ddea28cdf1a848"}
Oct 30 18:47:08.497909 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.497613    6014 scope.go:122] "RemoveContainer" containerID="33851451ba4fb97c51dd421327866fe269d8b6fbfec59d1d25bec8cc0a37089e"
Oct 30 18:47:08.506461 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.506438    6014 scope.go:122] "RemoveContainer" containerID="33851451ba4fb97c51dd421327866fe269d8b6fbfec59d1d25bec8cc0a37089e"
Oct 30 18:47:08.506754 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:47:08.506733    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33851451ba4fb97c51dd421327866fe269d8b6fbfec59d1d25bec8cc0a37089e\": container with ID starting with 33851451ba4fb97c51dd421327866fe269d8b6fbfec59d1d25bec8cc0a37089e not found: ID does not exist" containerID="33851451ba4fb97c51dd421327866fe269d8b6fbfec59d1d25bec8cc0a37089e"
Oct 30 18:47:08.506806 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.506764    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33851451ba4fb97c51dd421327866fe269d8b6fbfec59d1d25bec8cc0a37089e"} err="failed to get container status \"33851451ba4fb97c51dd421327866fe269d8b6fbfec59d1d25bec8cc0a37089e\": rpc error: code = NotFound desc = could not find container \"33851451ba4fb97c51dd421327866fe269d8b6fbfec59d1d25bec8cc0a37089e\": container with ID starting with 33851451ba4fb97c51dd421327866fe269d8b6fbfec59d1d25bec8cc0a37089e not found: ID does not exist"
Oct 30 18:47:08.668296 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.668267    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"]
Oct 30 18:47:08.674947 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:08.674924    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-gvz5j"]
Oct 30 18:47:09.474339 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:09.474309    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a73748f-c79c-4b86-b0be-33b914048cc2" path="/var/lib/kubelet/pods/0a73748f-c79c-4b86-b0be-33b914048cc2/volumes"
Oct 30 18:47:20.541919 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:20.541876    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"]
Oct 30 18:47:20.542432 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:20.542177    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz" podUID="7d4e71c5-5b67-4098-af2e-d5218b15ba47" containerName="manager" containerID="cri-o://75fa067235a1c85d7bb3e44839c4bd3e0fb7de42e558183501227194648f3b12" gracePeriod=10
Oct 30 18:47:20.798957 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:20.798902    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"
Oct 30 18:47:20.894354 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:20.894316    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d4e71c5-5b67-4098-af2e-d5218b15ba47-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d4e71c5-5b67-4098-af2e-d5218b15ba47-extensions-socket-volume\") pod \"7d4e71c5-5b67-4098-af2e-d5218b15ba47\" (UID: \"7d4e71c5-5b67-4098-af2e-d5218b15ba47\") "
Oct 30 18:47:20.894557 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:20.894400    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4e71c5-5b67-4098-af2e-d5218b15ba47-kube-api-access-266np\" (UniqueName: \"kubernetes.io/projected/7d4e71c5-5b67-4098-af2e-d5218b15ba47-kube-api-access-266np\") pod \"7d4e71c5-5b67-4098-af2e-d5218b15ba47\" (UID: \"7d4e71c5-5b67-4098-af2e-d5218b15ba47\") "
Oct 30 18:47:20.894699 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:20.894676    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4e71c5-5b67-4098-af2e-d5218b15ba47-extensions-socket-volume" pod "7d4e71c5-5b67-4098-af2e-d5218b15ba47" (UID: "7d4e71c5-5b67-4098-af2e-d5218b15ba47"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:47:20.896655 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:20.896633    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4e71c5-5b67-4098-af2e-d5218b15ba47-kube-api-access-266np" pod "7d4e71c5-5b67-4098-af2e-d5218b15ba47" (UID: "7d4e71c5-5b67-4098-af2e-d5218b15ba47"). InnerVolumeSpecName "kube-api-access-266np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:20.995676 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:20.995641    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-266np\" (UniqueName: \"kubernetes.io/projected/7d4e71c5-5b67-4098-af2e-d5218b15ba47-kube-api-access-266np\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:47:20.995676 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:20.995670    6014 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d4e71c5-5b67-4098-af2e-d5218b15ba47-extensions-socket-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:47:21.570771 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:21.570736    6014 generic.go:356] "Generic (PLEG): container finished" podID="7d4e71c5-5b67-4098-af2e-d5218b15ba47" containerID="75fa067235a1c85d7bb3e44839c4bd3e0fb7de42e558183501227194648f3b12" exitCode=0
Oct 30 18:47:21.571292 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:21.570780    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz" event={"ID":"7d4e71c5-5b67-4098-af2e-d5218b15ba47","Type":"ContainerDied","Data":"75fa067235a1c85d7bb3e44839c4bd3e0fb7de42e558183501227194648f3b12"}
Oct 30 18:47:21.571292 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:21.570806    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"
Oct 30 18:47:21.571292 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:21.570837    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz" event={"ID":"7d4e71c5-5b67-4098-af2e-d5218b15ba47","Type":"ContainerDied","Data":"9d02e7c1e74b43eee0c65f872860c29b93623d193e18ae56a8d976b88eb53969"}
Oct 30 18:47:21.571292 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:21.570862    6014 scope.go:122] "RemoveContainer" containerID="75fa067235a1c85d7bb3e44839c4bd3e0fb7de42e558183501227194648f3b12"
Oct 30 18:47:21.579186 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:21.579170    6014 scope.go:122] "RemoveContainer" containerID="75fa067235a1c85d7bb3e44839c4bd3e0fb7de42e558183501227194648f3b12"
Oct 30 18:47:21.579439 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:47:21.579421    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75fa067235a1c85d7bb3e44839c4bd3e0fb7de42e558183501227194648f3b12\": container with ID starting with 75fa067235a1c85d7bb3e44839c4bd3e0fb7de42e558183501227194648f3b12 not found: ID does not exist" containerID="75fa067235a1c85d7bb3e44839c4bd3e0fb7de42e558183501227194648f3b12"
Oct 30 18:47:21.579502 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:21.579461    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75fa067235a1c85d7bb3e44839c4bd3e0fb7de42e558183501227194648f3b12"} err="failed to get container status \"75fa067235a1c85d7bb3e44839c4bd3e0fb7de42e558183501227194648f3b12\": rpc error: code = NotFound desc = could not find container \"75fa067235a1c85d7bb3e44839c4bd3e0fb7de42e558183501227194648f3b12\": container with ID starting with 75fa067235a1c85d7bb3e44839c4bd3e0fb7de42e558183501227194648f3b12 not found: ID does not exist"
Oct 30 18:47:21.736947 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:21.736863    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"]
Oct 30 18:47:21.786831 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:21.786797    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-bfglz"]
Oct 30 18:47:23.258223 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:47:23.258184    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:47:23.480536 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:23.480503    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4e71c5-5b67-4098-af2e-d5218b15ba47" path="/var/lib/kubelet/pods/7d4e71c5-5b67-4098-af2e-d5218b15ba47/volumes"
Oct 30 18:47:25.328268 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.328236    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"]
Oct 30 18:47:25.328657 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.328608    6014 cpu_manager.go:414] "containerMap: removing container" podUID="7d4e71c5-5b67-4098-af2e-d5218b15ba47" containerName="manager"
Oct 30 18:47:25.328657 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.328620    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4e71c5-5b67-4098-af2e-d5218b15ba47" containerName="manager"
Oct 30 18:47:25.328657 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.328633    6014 cpu_manager.go:414] "containerMap: removing container" podUID="0a73748f-c79c-4b86-b0be-33b914048cc2" containerName="manager"
Oct 30 18:47:25.328657 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.328638    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a73748f-c79c-4b86-b0be-33b914048cc2" containerName="manager"
Oct 30 18:47:25.328780 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.328710    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4e71c5-5b67-4098-af2e-d5218b15ba47" containerName="manager"
Oct 30 18:47:25.328780 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.328719    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a73748f-c79c-4b86-b0be-33b914048cc2" containerName="manager"
Oct 30 18:47:25.331597 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.331575    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.386061 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.383537    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qwc8z\""
Oct 30 18:47:25.436809 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.436760    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.436809 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.436804    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.437184 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.436833    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.437184 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.436900    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.437184 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.437020    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cjdw\" (UniqueName: \"kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-kube-api-access-7cjdw\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.437184 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.437077    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.437184 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.437109    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.437402 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.437218    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.437402 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.437281    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.440983 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.440955    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"]
Oct 30 18:47:25.538195 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.538154    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cjdw\" (UniqueName: \"kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-kube-api-access-7cjdw\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.538413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.538204    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.538413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.538235    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.538413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.538273    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.538413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.538318    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.538413 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.538405    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.538747 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.538436    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.538747 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.538465    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.538747 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.538489    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.538747 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.538688    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.538921 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.538776    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.539032 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.538984    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.539316 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.539290    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.539405 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.539333    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.540886 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.540865    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.540998 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.540982    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.595333 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.595259    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.595333 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.595275    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cjdw\" (UniqueName: \"kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-kube-api-access-7cjdw\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jzvkj\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:25.643060 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:25.643021    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:26.067409 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:26.067376    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"]
Oct 30 18:47:26.071074 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:47:26.071044    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cdf1a74_fe4f_4e2e_bc7e_0f7d259e4a7a.slice/crio-17280558be478dcb22358716d8ca57c450397b5a605bab1964dd37767df3b6f1 WatchSource:0}: Error finding container 17280558be478dcb22358716d8ca57c450397b5a605bab1964dd37767df3b6f1: Status 404 returned error can't find the container with id 17280558be478dcb22358716d8ca57c450397b5a605bab1964dd37767df3b6f1
Oct 30 18:47:26.073427 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:26.073392    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:26.073514 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:26.073480    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:26.073514 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:26.073511    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:26.595668 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:26.595639    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" event={"ID":"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a","Type":"ContainerStarted","Data":"37dd2c746f28d42e081fa5dec673eeebc456a3f6c309995aac6073e281e920c3"}
Oct 30 18:47:26.596045 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:26.595693    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" event={"ID":"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a","Type":"ContainerStarted","Data":"17280558be478dcb22358716d8ca57c450397b5a605bab1964dd37767df3b6f1"}
Oct 30 18:47:26.643267 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:26.643230    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:26.739897 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:26.739845    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" podStartSLOduration=1.7398305280000002 podStartE2EDuration="1.739830528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:26.739317637 +0000 UTC m=+485.382485250" watchObservedRunningTime="2026-10-30 18:47:26.739830528 +0000 UTC m=+485.382998140"
Oct 30 18:47:27.648706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:27.648677    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:28.604565 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:28.604525    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:28.605475 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:28.605457    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:47:43.788180 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:43.788129    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m5d7r"]
Oct 30 18:47:43.790643 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:43.790627    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m5d7r"
Oct 30 18:47:43.842291 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:43.842255    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nv74j\""
Oct 30 18:47:43.892898 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:43.892866    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m5d7r"]
Oct 30 18:47:43.919886 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:43.919853    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlt2k\" (UniqueName: \"kubernetes.io/projected/c738ce1a-aea7-4238-afc7-4e92624c5bde-kube-api-access-vlt2k\") pod \"authorino-57f7f8659f-m5d7r\" (UID: \"c738ce1a-aea7-4238-afc7-4e92624c5bde\") " pod="kuadrant-system/authorino-57f7f8659f-m5d7r"
Oct 30 18:47:44.020899 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:44.020863    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlt2k\" (UniqueName: \"kubernetes.io/projected/c738ce1a-aea7-4238-afc7-4e92624c5bde-kube-api-access-vlt2k\") pod \"authorino-57f7f8659f-m5d7r\" (UID: \"c738ce1a-aea7-4238-afc7-4e92624c5bde\") " pod="kuadrant-system/authorino-57f7f8659f-m5d7r"
Oct 30 18:47:44.080570 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:44.080536    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlt2k\" (UniqueName: \"kubernetes.io/projected/c738ce1a-aea7-4238-afc7-4e92624c5bde-kube-api-access-vlt2k\") pod \"authorino-57f7f8659f-m5d7r\" (UID: \"c738ce1a-aea7-4238-afc7-4e92624c5bde\") " pod="kuadrant-system/authorino-57f7f8659f-m5d7r"
Oct 30 18:47:44.101950 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:44.101924    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m5d7r"
Oct 30 18:47:44.466520 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:44.466440    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m5d7r"]
Oct 30 18:47:44.470279 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:47:44.470244    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc738ce1a_aea7_4238_afc7_4e92624c5bde.slice/crio-4170e26541380e61502030ab27cd45d485863ff6be48d1d05696c90bc1e3c3b8 WatchSource:0}: Error finding container 4170e26541380e61502030ab27cd45d485863ff6be48d1d05696c90bc1e3c3b8: Status 404 returned error can't find the container with id 4170e26541380e61502030ab27cd45d485863ff6be48d1d05696c90bc1e3c3b8
Oct 30 18:47:44.676481 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:44.676443    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m5d7r" event={"ID":"c738ce1a-aea7-4238-afc7-4e92624c5bde","Type":"ContainerStarted","Data":"4170e26541380e61502030ab27cd45d485863ff6be48d1d05696c90bc1e3c3b8"}
Oct 30 18:47:45.347197 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:45.347162    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:47:45.350388 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:45.350368    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"
Oct 30 18:47:45.401799 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:45.401768    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hdxfm\""
Oct 30 18:47:45.433545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:45.433500    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1d97859-2437-49ca-9dc4-7e5d381b2e86-config-file\") pod \"limitador-limitador-7c9b87d574-bdl7l\" (UID: \"e1d97859-2437-49ca-9dc4-7e5d381b2e86\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"
Oct 30 18:47:45.433753 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:45.433585    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxcgw\" (UniqueName: \"kubernetes.io/projected/e1d97859-2437-49ca-9dc4-7e5d381b2e86-kube-api-access-bxcgw\") pod \"limitador-limitador-7c9b87d574-bdl7l\" (UID: \"e1d97859-2437-49ca-9dc4-7e5d381b2e86\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"
Oct 30 18:47:45.451319 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:45.451287    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 18:47:45.453149 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:45.453111    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:47:45.534567 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:45.534529    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxcgw\" (UniqueName: \"kubernetes.io/projected/e1d97859-2437-49ca-9dc4-7e5d381b2e86-kube-api-access-bxcgw\") pod \"limitador-limitador-7c9b87d574-bdl7l\" (UID: \"e1d97859-2437-49ca-9dc4-7e5d381b2e86\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"
Oct 30 18:47:45.534745 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:45.534616    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1d97859-2437-49ca-9dc4-7e5d381b2e86-config-file\") pod \"limitador-limitador-7c9b87d574-bdl7l\" (UID: \"e1d97859-2437-49ca-9dc4-7e5d381b2e86\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"
Oct 30 18:47:45.535246 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:45.535229    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1d97859-2437-49ca-9dc4-7e5d381b2e86-config-file\") pod \"limitador-limitador-7c9b87d574-bdl7l\" (UID: \"e1d97859-2437-49ca-9dc4-7e5d381b2e86\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"
Oct 30 18:47:45.590875 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:45.590845    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxcgw\" (UniqueName: \"kubernetes.io/projected/e1d97859-2437-49ca-9dc4-7e5d381b2e86-kube-api-access-bxcgw\") pod \"limitador-limitador-7c9b87d574-bdl7l\" (UID: \"e1d97859-2437-49ca-9dc4-7e5d381b2e86\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"
Oct 30 18:47:45.661997 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:45.661913    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"
Oct 30 18:47:45.666799 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:45.666771    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:47:46.035603 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:46.031519    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:47:46.039772 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:47:46.039743    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d97859_2437_49ca_9dc4_7e5d381b2e86.slice/crio-359524e3945f992e2afb13a9f72365acdc789a79ffcf051e00a44d00bea45ac6 WatchSource:0}: Error finding container 359524e3945f992e2afb13a9f72365acdc789a79ffcf051e00a44d00bea45ac6: Status 404 returned error can't find the container with id 359524e3945f992e2afb13a9f72365acdc789a79ffcf051e00a44d00bea45ac6
Oct 30 18:47:46.687334 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:46.687300    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-bdl7l" event={"ID":"e1d97859-2437-49ca-9dc4-7e5d381b2e86","Type":"ContainerStarted","Data":"359524e3945f992e2afb13a9f72365acdc789a79ffcf051e00a44d00bea45ac6"}
Oct 30 18:47:47.695211 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:47.695173    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m5d7r" event={"ID":"c738ce1a-aea7-4238-afc7-4e92624c5bde","Type":"ContainerStarted","Data":"4c3cd2cb7e5fcb293efd6a140f45a15192c5dc23a66800984314d80395ef89b5"}
Oct 30 18:47:47.807532 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:47.806476    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-m5d7r" podStartSLOduration=1.994682911 podStartE2EDuration="4.806457705s" totalImagesPullingTime="2.811774794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:47.805874462 +0000 UTC m=+506.449042076" watchObservedRunningTime="2026-10-30 18:47:47.806457705 +0000 UTC m=+506.449625320"
Oct 30 18:47:48.700650 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:48.700607    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-bdl7l" event={"ID":"e1d97859-2437-49ca-9dc4-7e5d381b2e86","Type":"ContainerStarted","Data":"3d1782a01bc6faa38b4e26a653cec9cf1e766c499cd2c1da240eb458b0e03a8c"}
Oct 30 18:47:48.815936 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:48.815879    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-bdl7l" podStartSLOduration=1.8683710040000001 podStartE2EDuration="3.815861698s" totalImagesPullingTime="1.947490694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:48.815311975 +0000 UTC m=+507.458479591" watchObservedRunningTime="2026-10-30 18:47:48.815861698 +0000 UTC m=+507.459029345"
Oct 30 18:47:49.603381 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:49.603344    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m5d7r"]
Oct 30 18:47:49.705667 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:49.705621    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-m5d7r" podUID="c738ce1a-aea7-4238-afc7-4e92624c5bde" containerName="authorino" containerID="cri-o://4c3cd2cb7e5fcb293efd6a140f45a15192c5dc23a66800984314d80395ef89b5" gracePeriod=30
Oct 30 18:47:49.706114 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:49.705816    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"
Oct 30 18:47:49.954767 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:49.954743    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m5d7r"
Oct 30 18:47:50.078341 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:50.078306    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c738ce1a-aea7-4238-afc7-4e92624c5bde-kube-api-access-vlt2k\" (UniqueName: \"kubernetes.io/projected/c738ce1a-aea7-4238-afc7-4e92624c5bde-kube-api-access-vlt2k\") pod \"c738ce1a-aea7-4238-afc7-4e92624c5bde\" (UID: \"c738ce1a-aea7-4238-afc7-4e92624c5bde\") "
Oct 30 18:47:50.080814 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:50.080778    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c738ce1a-aea7-4238-afc7-4e92624c5bde-kube-api-access-vlt2k" pod "c738ce1a-aea7-4238-afc7-4e92624c5bde" (UID: "c738ce1a-aea7-4238-afc7-4e92624c5bde"). InnerVolumeSpecName "kube-api-access-vlt2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:50.179663 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:50.179577    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlt2k\" (UniqueName: \"kubernetes.io/projected/c738ce1a-aea7-4238-afc7-4e92624c5bde-kube-api-access-vlt2k\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:47:50.711509 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:50.711473    6014 generic.go:356] "Generic (PLEG): container finished" podID="c738ce1a-aea7-4238-afc7-4e92624c5bde" containerID="4c3cd2cb7e5fcb293efd6a140f45a15192c5dc23a66800984314d80395ef89b5" exitCode=0
Oct 30 18:47:50.712002 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:50.711518    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m5d7r"
Oct 30 18:47:50.712002 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:50.711567    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m5d7r" event={"ID":"c738ce1a-aea7-4238-afc7-4e92624c5bde","Type":"ContainerDied","Data":"4c3cd2cb7e5fcb293efd6a140f45a15192c5dc23a66800984314d80395ef89b5"}
Oct 30 18:47:50.712002 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:50.711608    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m5d7r" event={"ID":"c738ce1a-aea7-4238-afc7-4e92624c5bde","Type":"ContainerDied","Data":"4170e26541380e61502030ab27cd45d485863ff6be48d1d05696c90bc1e3c3b8"}
Oct 30 18:47:50.712002 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:50.711624    6014 scope.go:122] "RemoveContainer" containerID="4c3cd2cb7e5fcb293efd6a140f45a15192c5dc23a66800984314d80395ef89b5"
Oct 30 18:47:50.720532 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:50.720515    6014 scope.go:122] "RemoveContainer" containerID="4c3cd2cb7e5fcb293efd6a140f45a15192c5dc23a66800984314d80395ef89b5"
Oct 30 18:47:50.720823 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:47:50.720802    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3cd2cb7e5fcb293efd6a140f45a15192c5dc23a66800984314d80395ef89b5\": container with ID starting with 4c3cd2cb7e5fcb293efd6a140f45a15192c5dc23a66800984314d80395ef89b5 not found: ID does not exist" containerID="4c3cd2cb7e5fcb293efd6a140f45a15192c5dc23a66800984314d80395ef89b5"
Oct 30 18:47:50.720903 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:50.720830    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3cd2cb7e5fcb293efd6a140f45a15192c5dc23a66800984314d80395ef89b5"} err="failed to get container status \"4c3cd2cb7e5fcb293efd6a140f45a15192c5dc23a66800984314d80395ef89b5\": rpc error: code = NotFound desc = could not find container \"4c3cd2cb7e5fcb293efd6a140f45a15192c5dc23a66800984314d80395ef89b5\": container with ID starting with 4c3cd2cb7e5fcb293efd6a140f45a15192c5dc23a66800984314d80395ef89b5 not found: ID does not exist"
Oct 30 18:47:50.871966 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:50.871935    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m5d7r"]
Oct 30 18:47:50.875718 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:50.875696    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m5d7r"]
Oct 30 18:47:51.475041 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:47:51.475009    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c738ce1a-aea7-4238-afc7-4e92624c5bde" path="/var/lib/kubelet/pods/c738ce1a-aea7-4238-afc7-4e92624c5bde/volumes"
Oct 30 18:47:53.257235 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:47:53.257198    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:48:00.713171 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:00.713122    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"
Oct 30 18:48:21.216623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.216584    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65fb4c857f-clv28"]
Oct 30 18:48:21.220291 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.217189    6014 cpu_manager.go:414] "containerMap: removing container" podUID="c738ce1a-aea7-4238-afc7-4e92624c5bde" containerName="authorino"
Oct 30 18:48:21.220291 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.217209    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c738ce1a-aea7-4238-afc7-4e92624c5bde" containerName="authorino"
Oct 30 18:48:21.220291 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.217383    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="c738ce1a-aea7-4238-afc7-4e92624c5bde" containerName="authorino"
Oct 30 18:48:21.221407 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.221392    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65fb4c857f-clv28"
Oct 30 18:48:21.273204 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.273172    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65fb4c857f-clv28"]
Oct 30 18:48:21.321523 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.321491    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nv74j\""
Oct 30 18:48:21.377123 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.377089    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc58q\" (UniqueName: \"kubernetes.io/projected/51206f6b-f35e-4273-a3f3-2d55eeecf47b-kube-api-access-vc58q\") pod \"authorino-65fb4c857f-clv28\" (UID: \"51206f6b-f35e-4273-a3f3-2d55eeecf47b\") " pod="kuadrant-system/authorino-65fb4c857f-clv28"
Oct 30 18:48:21.478449 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.478370    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc58q\" (UniqueName: \"kubernetes.io/projected/51206f6b-f35e-4273-a3f3-2d55eeecf47b-kube-api-access-vc58q\") pod \"authorino-65fb4c857f-clv28\" (UID: \"51206f6b-f35e-4273-a3f3-2d55eeecf47b\") " pod="kuadrant-system/authorino-65fb4c857f-clv28"
Oct 30 18:48:21.535570 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.535540    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc58q\" (UniqueName: \"kubernetes.io/projected/51206f6b-f35e-4273-a3f3-2d55eeecf47b-kube-api-access-vc58q\") pod \"authorino-65fb4c857f-clv28\" (UID: \"51206f6b-f35e-4273-a3f3-2d55eeecf47b\") " pod="kuadrant-system/authorino-65fb4c857f-clv28"
Oct 30 18:48:21.674065 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.674028    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65fb4c857f-clv28"]
Oct 30 18:48:21.705296 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.705261    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7978c8786f-hn7br"]
Oct 30 18:48:21.711897 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.711874    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7978c8786f-hn7br"
Oct 30 18:48:21.725089 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.725062    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nv74j\""
Oct 30 18:48:21.735236 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.735182    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65fb4c857f-clv28"
Oct 30 18:48:21.764351 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.764322    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 18:48:21.765039 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.765009    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7978c8786f-hn7br"]
Oct 30 18:48:21.780295 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.780266    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b5234541-83d3-4952-bce1-498435d99e3d-tls-cert\") pod \"authorino-7978c8786f-hn7br\" (UID: \"b5234541-83d3-4952-bce1-498435d99e3d\") " pod="kuadrant-system/authorino-7978c8786f-hn7br"
Oct 30 18:48:21.780459 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.780431    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hth56\" (UniqueName: \"kubernetes.io/projected/b5234541-83d3-4952-bce1-498435d99e3d-kube-api-access-hth56\") pod \"authorino-7978c8786f-hn7br\" (UID: \"b5234541-83d3-4952-bce1-498435d99e3d\") " pod="kuadrant-system/authorino-7978c8786f-hn7br"
Oct 30 18:48:21.881725 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.881689    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b5234541-83d3-4952-bce1-498435d99e3d-tls-cert\") pod \"authorino-7978c8786f-hn7br\" (UID: \"b5234541-83d3-4952-bce1-498435d99e3d\") " pod="kuadrant-system/authorino-7978c8786f-hn7br"
Oct 30 18:48:21.881929 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.881781    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hth56\" (UniqueName: \"kubernetes.io/projected/b5234541-83d3-4952-bce1-498435d99e3d-kube-api-access-hth56\") pod \"authorino-7978c8786f-hn7br\" (UID: \"b5234541-83d3-4952-bce1-498435d99e3d\") " pod="kuadrant-system/authorino-7978c8786f-hn7br"
Oct 30 18:48:21.884486 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.884464    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b5234541-83d3-4952-bce1-498435d99e3d-tls-cert\") pod \"authorino-7978c8786f-hn7br\" (UID: \"b5234541-83d3-4952-bce1-498435d99e3d\") " pod="kuadrant-system/authorino-7978c8786f-hn7br"
Oct 30 18:48:21.940421 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:21.940392    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hth56\" (UniqueName: \"kubernetes.io/projected/b5234541-83d3-4952-bce1-498435d99e3d-kube-api-access-hth56\") pod \"authorino-7978c8786f-hn7br\" (UID: \"b5234541-83d3-4952-bce1-498435d99e3d\") " pod="kuadrant-system/authorino-7978c8786f-hn7br"
Oct 30 18:48:22.024200 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:22.024092    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7978c8786f-hn7br"
Oct 30 18:48:22.110261 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:22.110236    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65fb4c857f-clv28"]
Oct 30 18:48:22.112831 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:48:22.112802    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51206f6b_f35e_4273_a3f3_2d55eeecf47b.slice/crio-07fe630063adfada14165d75bc6eb2a5bcb2335d12a77e7362111222d234ab33 WatchSource:0}: Error finding container 07fe630063adfada14165d75bc6eb2a5bcb2335d12a77e7362111222d234ab33: Status 404 returned error can't find the container with id 07fe630063adfada14165d75bc6eb2a5bcb2335d12a77e7362111222d234ab33
Oct 30 18:48:22.114172 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:22.114153    6014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:48:22.394897 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:22.394864    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7978c8786f-hn7br"]
Oct 30 18:48:22.398083 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:48:22.398055    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5234541_83d3_4952_bce1_498435d99e3d.slice/crio-9fa96cc7d2dc02e970e170ebcc002943c8b67018f33c44d2da47f8a6cb4b9d6e WatchSource:0}: Error finding container 9fa96cc7d2dc02e970e170ebcc002943c8b67018f33c44d2da47f8a6cb4b9d6e: Status 404 returned error can't find the container with id 9fa96cc7d2dc02e970e170ebcc002943c8b67018f33c44d2da47f8a6cb4b9d6e
Oct 30 18:48:22.861395 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:22.861361    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7978c8786f-hn7br" event={"ID":"b5234541-83d3-4952-bce1-498435d99e3d","Type":"ContainerStarted","Data":"9fa96cc7d2dc02e970e170ebcc002943c8b67018f33c44d2da47f8a6cb4b9d6e"}
Oct 30 18:48:22.862524 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:22.862498    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65fb4c857f-clv28" event={"ID":"51206f6b-f35e-4273-a3f3-2d55eeecf47b","Type":"ContainerStarted","Data":"07fe630063adfada14165d75bc6eb2a5bcb2335d12a77e7362111222d234ab33"}
Oct 30 18:48:23.256838 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:48:23.256774    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:48:23.869551 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:23.869512    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65fb4c857f-clv28" event={"ID":"51206f6b-f35e-4273-a3f3-2d55eeecf47b","Type":"ContainerStarted","Data":"07f3b2c400dc2ce46233a61da81d7040d367842a7465bed0c6baf0ee9887b1c2"}
Oct 30 18:48:23.870063 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:23.869587    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-65fb4c857f-clv28" podUID="51206f6b-f35e-4273-a3f3-2d55eeecf47b" containerName="authorino" containerID="cri-o://07f3b2c400dc2ce46233a61da81d7040d367842a7465bed0c6baf0ee9887b1c2" gracePeriod=30
Oct 30 18:48:23.871290 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:23.871264    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7978c8786f-hn7br" event={"ID":"b5234541-83d3-4952-bce1-498435d99e3d","Type":"ContainerStarted","Data":"374b14e06512932968b68f95db576b7c4b9ab422a67adeeb1e02418c8c1d0dfe"}
Oct 30 18:48:23.984851 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:23.984754    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-65fb4c857f-clv28" podStartSLOduration=1.713361149 podStartE2EDuration="2.984740925s" totalImagesPullingTime="1.271379776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:23.98380551 +0000 UTC m=+542.626973125" watchObservedRunningTime="2026-10-30 18:48:23.984740925 +0000 UTC m=+542.627908538"
Oct 30 18:48:24.099585 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:24.099527    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7978c8786f-hn7br" podStartSLOduration=1.8589364860000002 podStartE2EDuration="3.099511212s" totalImagesPullingTime="1.240574726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:24.098393756 +0000 UTC m=+542.741561370" watchObservedRunningTime="2026-10-30 18:48:24.099511212 +0000 UTC m=+542.742678825"
Oct 30 18:48:24.126590 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:24.126558    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65fb4c857f-clv28"
Oct 30 18:48:24.205247 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:24.205218    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51206f6b-f35e-4273-a3f3-2d55eeecf47b-kube-api-access-vc58q\" (UniqueName: \"kubernetes.io/projected/51206f6b-f35e-4273-a3f3-2d55eeecf47b-kube-api-access-vc58q\") pod \"51206f6b-f35e-4273-a3f3-2d55eeecf47b\" (UID: \"51206f6b-f35e-4273-a3f3-2d55eeecf47b\") "
Oct 30 18:48:24.207539 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:24.207510    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51206f6b-f35e-4273-a3f3-2d55eeecf47b-kube-api-access-vc58q" pod "51206f6b-f35e-4273-a3f3-2d55eeecf47b" (UID: "51206f6b-f35e-4273-a3f3-2d55eeecf47b"). InnerVolumeSpecName "kube-api-access-vc58q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:24.306524 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:24.306443    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc58q\" (UniqueName: \"kubernetes.io/projected/51206f6b-f35e-4273-a3f3-2d55eeecf47b-kube-api-access-vc58q\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:48:24.877794 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:24.877758    6014 generic.go:356] "Generic (PLEG): container finished" podID="51206f6b-f35e-4273-a3f3-2d55eeecf47b" containerID="07f3b2c400dc2ce46233a61da81d7040d367842a7465bed0c6baf0ee9887b1c2" exitCode=0
Oct 30 18:48:24.878343 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:24.877816    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65fb4c857f-clv28"
Oct 30 18:48:24.878343 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:24.877831    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65fb4c857f-clv28" event={"ID":"51206f6b-f35e-4273-a3f3-2d55eeecf47b","Type":"ContainerDied","Data":"07f3b2c400dc2ce46233a61da81d7040d367842a7465bed0c6baf0ee9887b1c2"}
Oct 30 18:48:24.878343 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:24.877890    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65fb4c857f-clv28" event={"ID":"51206f6b-f35e-4273-a3f3-2d55eeecf47b","Type":"ContainerDied","Data":"07fe630063adfada14165d75bc6eb2a5bcb2335d12a77e7362111222d234ab33"}
Oct 30 18:48:24.878343 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:24.877921    6014 scope.go:122] "RemoveContainer" containerID="07f3b2c400dc2ce46233a61da81d7040d367842a7465bed0c6baf0ee9887b1c2"
Oct 30 18:48:24.886823 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:24.886797    6014 scope.go:122] "RemoveContainer" containerID="07f3b2c400dc2ce46233a61da81d7040d367842a7465bed0c6baf0ee9887b1c2"
Oct 30 18:48:24.887121 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:48:24.887102    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f3b2c400dc2ce46233a61da81d7040d367842a7465bed0c6baf0ee9887b1c2\": container with ID starting with 07f3b2c400dc2ce46233a61da81d7040d367842a7465bed0c6baf0ee9887b1c2 not found: ID does not exist" containerID="07f3b2c400dc2ce46233a61da81d7040d367842a7465bed0c6baf0ee9887b1c2"
Oct 30 18:48:24.887223 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:24.887132    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f3b2c400dc2ce46233a61da81d7040d367842a7465bed0c6baf0ee9887b1c2"} err="failed to get container status \"07f3b2c400dc2ce46233a61da81d7040d367842a7465bed0c6baf0ee9887b1c2\": rpc error: code = NotFound desc = could not find container \"07f3b2c400dc2ce46233a61da81d7040d367842a7465bed0c6baf0ee9887b1c2\": container with ID starting with 07f3b2c400dc2ce46233a61da81d7040d367842a7465bed0c6baf0ee9887b1c2 not found: ID does not exist"
Oct 30 18:48:25.043304 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:25.043223    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65fb4c857f-clv28"]
Oct 30 18:48:25.046378 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:25.046355    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65fb4c857f-clv28"]
Oct 30 18:48:25.475125 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:48:25.475094    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51206f6b-f35e-4273-a3f3-2d55eeecf47b" path="/var/lib/kubelet/pods/51206f6b-f35e-4273-a3f3-2d55eeecf47b/volumes"
Oct 30 18:48:53.256701 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:48:53.256619    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:49:15.000736 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:15.000692    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:49:23.258637 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:49:23.258585    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:49:24.158061 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:24.158028    6014 generic.go:356] "Generic (PLEG): container finished" podID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerID="37dd2c746f28d42e081fa5dec673eeebc456a3f6c309995aac6073e281e920c3" exitCode=0
Oct 30 18:49:24.158061 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:24.158070    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" event={"ID":"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a","Type":"ContainerDied","Data":"37dd2c746f28d42e081fa5dec673eeebc456a3f6c309995aac6073e281e920c3"}
Oct 30 18:49:24.158478 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:24.158465    6014 scope.go:122] "RemoveContainer" containerID="37dd2c746f28d42e081fa5dec673eeebc456a3f6c309995aac6073e281e920c3"
Oct 30 18:49:24.159531 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:24.159501    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:24.159636 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:24.159587    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:24.159698 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:24.159662    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:25.178019 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:25.177981    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" event={"ID":"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a","Type":"ContainerStarted","Data":"222fb88bed456a87e5fb009c9decd9e97f4cb913b6f4e1d56c80af97e28fc66f"}
Oct 30 18:49:25.643227 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:25.643192    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:49:25.644381 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:25.644346    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jzvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 30 18:49:25.644507 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:25.644406    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 30 18:49:26.643450 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:26.643413    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jzvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 30 18:49:26.643860 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:26.643495    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 30 18:49:27.643552 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:27.643512    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jzvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 30 18:49:27.643971 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:27.643563    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 30 18:49:28.604912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:28.604874    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:49:28.643788 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:28.643761    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jzvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 30 18:49:28.644167 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:28.643815    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 30 18:49:29.643408 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:29.643369    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jzvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 30 18:49:29.643599 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:29.643426    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 30 18:49:30.647486 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:30.647448    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:49:30.648373 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:49:30.648357    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:49:53.256583 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:49:53.256545    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:50:18.793305 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:18.793209    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:50:22.722054 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.722018    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"]
Oct 30 18:50:22.722479 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.722467    6014 cpu_manager.go:414] "containerMap: removing container" podUID="51206f6b-f35e-4273-a3f3-2d55eeecf47b" containerName="authorino"
Oct 30 18:50:22.722521 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.722480    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51206f6b-f35e-4273-a3f3-2d55eeecf47b" containerName="authorino"
Oct 30 18:50:22.722555 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.722534    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="51206f6b-f35e-4273-a3f3-2d55eeecf47b" containerName="authorino"
Oct 30 18:50:22.725797 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.725778    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.777760 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.777724    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 18:50:22.805769 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.805732    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5596ab6c-2bae-460e-81c1-e875ce418e88-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.806064 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.806033    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5596ab6c-2bae-460e-81c1-e875ce418e88-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.806206 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.806079    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stkgq\" (UniqueName: \"kubernetes.io/projected/5596ab6c-2bae-460e-81c1-e875ce418e88-kube-api-access-stkgq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.806206 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.806117    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5596ab6c-2bae-460e-81c1-e875ce418e88-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.806206 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.806190    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5596ab6c-2bae-460e-81c1-e875ce418e88-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.806554 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.806246    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5596ab6c-2bae-460e-81c1-e875ce418e88-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.827165 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.827081    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 18:50:22.827165 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.827081    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ntlts\""
Oct 30 18:50:22.827165 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.827132    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 18:50:22.828659 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.828635    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"]
Oct 30 18:50:22.906795 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.906765    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5596ab6c-2bae-460e-81c1-e875ce418e88-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.907007 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.906818    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5596ab6c-2bae-460e-81c1-e875ce418e88-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.907007 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.906860    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5596ab6c-2bae-460e-81c1-e875ce418e88-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.907007 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.906881    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stkgq\" (UniqueName: \"kubernetes.io/projected/5596ab6c-2bae-460e-81c1-e875ce418e88-kube-api-access-stkgq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.907007 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.906924    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5596ab6c-2bae-460e-81c1-e875ce418e88-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.907007 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.906983    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5596ab6c-2bae-460e-81c1-e875ce418e88-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.907418 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.907357    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5596ab6c-2bae-460e-81c1-e875ce418e88-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.907979 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.907570    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5596ab6c-2bae-460e-81c1-e875ce418e88-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.907979 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.907916    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5596ab6c-2bae-460e-81c1-e875ce418e88-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.910209 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.910185    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5596ab6c-2bae-460e-81c1-e875ce418e88-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.910397 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.910375    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5596ab6c-2bae-460e-81c1-e875ce418e88-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:22.965404 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:22.965322    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stkgq\" (UniqueName: \"kubernetes.io/projected/5596ab6c-2bae-460e-81c1-e875ce418e88-kube-api-access-stkgq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c\" (UID: \"5596ab6c-2bae-460e-81c1-e875ce418e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:23.038543 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:23.038499    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:23.260443 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:50:23.260341    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:50:23.422580 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:23.422548    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"]
Oct 30 18:50:23.426711 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:50:23.426678    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5596ab6c_2bae_460e_81c1_e875ce418e88.slice/crio-a9a04c08c87cc40a9996ceaa303c5352a39a3b12187e6ec3df2c19e8d21ac015 WatchSource:0}: Error finding container a9a04c08c87cc40a9996ceaa303c5352a39a3b12187e6ec3df2c19e8d21ac015: Status 404 returned error can't find the container with id a9a04c08c87cc40a9996ceaa303c5352a39a3b12187e6ec3df2c19e8d21ac015
Oct 30 18:50:23.441974 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:23.441945    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c" event={"ID":"5596ab6c-2bae-460e-81c1-e875ce418e88","Type":"ContainerStarted","Data":"a9a04c08c87cc40a9996ceaa303c5352a39a3b12187e6ec3df2c19e8d21ac015"}
Oct 30 18:50:25.455896 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:25.455860    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c" event={"ID":"5596ab6c-2bae-460e-81c1-e875ce418e88","Type":"ContainerStarted","Data":"e9888c885b4d5d6f11d47fa719ab25fd13abc79fe6c3a88e882299c187c666db"}
Oct 30 18:50:25.572583 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:25.572503    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c" podStartSLOduration=2.055925953 podStartE2EDuration="3.572485591s" totalImagesPullingTime="1.516559638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:25.571953343 +0000 UTC m=+664.215120949" watchObservedRunningTime="2026-10-30 18:50:25.572485591 +0000 UTC m=+664.215653208"
Oct 30 18:50:26.039615 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:26.039577    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:26.851543 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:26.851511    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:50:33.038859 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.038821    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:33.053884 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.053847    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:33.505606 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.505578    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c"
Oct 30 18:50:33.690726 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.690692    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"]
Oct 30 18:50:33.694316 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.694291    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.746304 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.746278    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 18:50:33.755952 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.755858    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"]
Oct 30 18:50:33.817131 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.817086    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7fbb25-1611-4efd-83fb-c636fdc232ff-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.817341 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.817167    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7fbb25-1611-4efd-83fb-c636fdc232ff-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.817341 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.817199    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxbxf\" (UniqueName: \"kubernetes.io/projected/2d7fbb25-1611-4efd-83fb-c636fdc232ff-kube-api-access-dxbxf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.817341 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.817249    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7fbb25-1611-4efd-83fb-c636fdc232ff-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.817446 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.817340    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7fbb25-1611-4efd-83fb-c636fdc232ff-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.817446 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.817372    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7fbb25-1611-4efd-83fb-c636fdc232ff-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.918772 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.918736    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7fbb25-1611-4efd-83fb-c636fdc232ff-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.918772 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.918779    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxbxf\" (UniqueName: \"kubernetes.io/projected/2d7fbb25-1611-4efd-83fb-c636fdc232ff-kube-api-access-dxbxf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.918988 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.918807    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7fbb25-1611-4efd-83fb-c636fdc232ff-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.918988 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.918869    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7fbb25-1611-4efd-83fb-c636fdc232ff-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.918988 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.918900    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7fbb25-1611-4efd-83fb-c636fdc232ff-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.919127 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.919007    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7fbb25-1611-4efd-83fb-c636fdc232ff-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.919229 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.919212    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7fbb25-1611-4efd-83fb-c636fdc232ff-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.919291 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.919266    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7fbb25-1611-4efd-83fb-c636fdc232ff-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.919458 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.919426    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7fbb25-1611-4efd-83fb-c636fdc232ff-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.921415 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.921389    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7fbb25-1611-4efd-83fb-c636fdc232ff-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.921605 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.921589    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7fbb25-1611-4efd-83fb-c636fdc232ff-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:33.975804 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:33.975778    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxbxf\" (UniqueName: \"kubernetes.io/projected/2d7fbb25-1611-4efd-83fb-c636fdc232ff-kube-api-access-dxbxf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67kkz\" (UID: \"2d7fbb25-1611-4efd-83fb-c636fdc232ff\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:34.006729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:34.006649    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:34.381560 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:34.381526    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"]
Oct 30 18:50:34.385286 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:50:34.385259    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7fbb25_1611_4efd_83fb_c636fdc232ff.slice/crio-55699c288eff433ba68107c7a8b956eb8e4b5354946bc2a0ae4afb55364892cd WatchSource:0}: Error finding container 55699c288eff433ba68107c7a8b956eb8e4b5354946bc2a0ae4afb55364892cd: Status 404 returned error can't find the container with id 55699c288eff433ba68107c7a8b956eb8e4b5354946bc2a0ae4afb55364892cd
Oct 30 18:50:34.500412 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:34.500372    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz" event={"ID":"2d7fbb25-1611-4efd-83fb-c636fdc232ff","Type":"ContainerStarted","Data":"55699c288eff433ba68107c7a8b956eb8e4b5354946bc2a0ae4afb55364892cd"}
Oct 30 18:50:35.506757 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:35.506718    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz" event={"ID":"2d7fbb25-1611-4efd-83fb-c636fdc232ff","Type":"ContainerStarted","Data":"34225e90bc2e4d44dca85d9a33d797c703a900b7eee758f8cfe690d8b61a0987"}
Oct 30 18:50:35.621114 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:35.621061    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz" podStartSLOduration=2.029727576 podStartE2EDuration="2.621046315s" totalImagesPullingTime="591.318739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:35.619884247 +0000 UTC m=+674.263051875" watchObservedRunningTime="2026-10-30 18:50:35.621046315 +0000 UTC m=+674.264213929"
Oct 30 18:50:36.007370 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:36.007342    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:36.221348 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:36.221314    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:50:37.165923 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.165884    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"]
Oct 30 18:50:37.170492 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.170467    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.254331 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.254294    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 18:50:37.277729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.277695    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"]
Oct 30 18:50:37.368492 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.368458    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.368691 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.368533    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kmv2\" (UniqueName: \"kubernetes.io/projected/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-kube-api-access-5kmv2\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.368691 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.368665    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.368772 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.368750    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.368834 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.368810    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.368956 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.368939    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.470097 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.469993    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.470097 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.470082    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.470366 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.470108    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.470366 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.470241    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.470366 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.470276    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.470366 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.470342    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kmv2\" (UniqueName: \"kubernetes.io/projected/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-kube-api-access-5kmv2\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.470599 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.470432    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.471037 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.471007    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.471180 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.471047    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.473800 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.473773    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.474439 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.474414    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.527324 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.527297    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kmv2\" (UniqueName: \"kubernetes.io/projected/e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208-kube-api-access-5kmv2\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z6rtg\" (UID: \"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:37.784984 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:37.784873    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:38.162610 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:38.162579    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"]
Oct 30 18:50:38.165784 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:50:38.165760    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7694c0f_a6d7_4a3a_bbd8_b2d59bce2208.slice/crio-1e272d57fcbf5b28826543d1aeb3271903e0bc7a013db0d0bc27f05a428c8d6a WatchSource:0}: Error finding container 1e272d57fcbf5b28826543d1aeb3271903e0bc7a013db0d0bc27f05a428c8d6a: Status 404 returned error can't find the container with id 1e272d57fcbf5b28826543d1aeb3271903e0bc7a013db0d0bc27f05a428c8d6a
Oct 30 18:50:38.522654 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:38.522551    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg" event={"ID":"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208","Type":"ContainerStarted","Data":"1e272d57fcbf5b28826543d1aeb3271903e0bc7a013db0d0bc27f05a428c8d6a"}
Oct 30 18:50:39.528249 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:39.528211    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg" event={"ID":"e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208","Type":"ContainerStarted","Data":"6c4d6f1ba4e2aa1316fe5e0ecdb1a33650f6a870e434a62087f005d5ebf815b3"}
Oct 30 18:50:39.666564 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:39.666499    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg" podStartSLOduration=2.046551791 podStartE2EDuration="2.666482972s" totalImagesPullingTime="619.931181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:39.66480851 +0000 UTC m=+678.307976125" watchObservedRunningTime="2026-10-30 18:50:39.666482972 +0000 UTC m=+678.309650585"
Oct 30 18:50:39.786030 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:39.785949    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:42.297590 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.297552    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"]
Oct 30 18:50:42.301758 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.301731    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.312079 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.312052    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79f5eaf5-cfd1-44df-878c-70c7798c2c12-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.313508 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.313486    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2nfg\" (UniqueName: \"kubernetes.io/projected/79f5eaf5-cfd1-44df-878c-70c7798c2c12-kube-api-access-w2nfg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.313680 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.313664    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79f5eaf5-cfd1-44df-878c-70c7798c2c12-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.313812 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.313796    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79f5eaf5-cfd1-44df-878c-70c7798c2c12-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.313989 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.313973    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79f5eaf5-cfd1-44df-878c-70c7798c2c12-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.314110 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.314096    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f5eaf5-cfd1-44df-878c-70c7798c2c12-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.355536 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.355504    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 18:50:42.406565 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.406530    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"]
Oct 30 18:50:42.416211 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.415709    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79f5eaf5-cfd1-44df-878c-70c7798c2c12-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.416211 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.415766    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2nfg\" (UniqueName: \"kubernetes.io/projected/79f5eaf5-cfd1-44df-878c-70c7798c2c12-kube-api-access-w2nfg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.416211 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.415821    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79f5eaf5-cfd1-44df-878c-70c7798c2c12-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.416211 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.415884    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79f5eaf5-cfd1-44df-878c-70c7798c2c12-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.416211 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.415981    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79f5eaf5-cfd1-44df-878c-70c7798c2c12-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.416211 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.416021    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f5eaf5-cfd1-44df-878c-70c7798c2c12-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.416792 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.416764    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79f5eaf5-cfd1-44df-878c-70c7798c2c12-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.417033 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.417012    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79f5eaf5-cfd1-44df-878c-70c7798c2c12-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.417419 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.417389    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79f5eaf5-cfd1-44df-878c-70c7798c2c12-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.419858 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.419831    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79f5eaf5-cfd1-44df-878c-70c7798c2c12-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.420253 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.420233    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f5eaf5-cfd1-44df-878c-70c7798c2c12-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.472905 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.472871    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2nfg\" (UniqueName: \"kubernetes.io/projected/79f5eaf5-cfd1-44df-878c-70c7798c2c12-kube-api-access-w2nfg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jg64x\" (UID: \"79f5eaf5-cfd1-44df-878c-70c7798c2c12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:42.622542 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:42.622510    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:43.004091 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:43.003988    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"]
Oct 30 18:50:43.009432 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:50:43.009390    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f5eaf5_cfd1_44df_878c_70c7798c2c12.slice/crio-b49f30c65beb6059268a892cbf1d3add4209418bb524651d75c95c0e1fd85644 WatchSource:0}: Error finding container b49f30c65beb6059268a892cbf1d3add4209418bb524651d75c95c0e1fd85644: Status 404 returned error can't find the container with id b49f30c65beb6059268a892cbf1d3add4209418bb524651d75c95c0e1fd85644
Oct 30 18:50:43.547901 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:43.547861    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x" event={"ID":"79f5eaf5-cfd1-44df-878c-70c7798c2c12","Type":"ContainerStarted","Data":"b49f30c65beb6059268a892cbf1d3add4209418bb524651d75c95c0e1fd85644"}
Oct 30 18:50:44.007114 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:44.007081    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:44.019647 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:44.019618    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:44.553872 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:44.553828    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x" event={"ID":"79f5eaf5-cfd1-44df-878c-70c7798c2c12","Type":"ContainerStarted","Data":"43d69439cb30ca752d3310b0bf7e6d735b0f236eda3d690948a4d8ec436b2666"}
Oct 30 18:50:44.565390 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:44.565369    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 18:50:44.622942 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:44.622909    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:44.671249 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:44.671177    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x" podStartSLOduration=2.061046848 podStartE2EDuration="2.671156964s" totalImagesPullingTime="610.110116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:44.670372275 +0000 UTC m=+683.313539890" watchObservedRunningTime="2026-10-30 18:50:44.671156964 +0000 UTC m=+683.314324591"
Oct 30 18:50:46.541323 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:46.541292    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:50:47.785999 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:47.785958    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:47.804876 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:47.804845    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:48.586937 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:48.586910    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg"
Oct 30 18:50:52.623280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:52.623248    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:52.636045 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:52.636016    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:53.257058 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:50:53.257014    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:50:53.610512 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:53.610477    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x"
Oct 30 18:50:55.820160 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:55.820102    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"]
Oct 30 18:50:55.823975 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:55.823957    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:55.927244 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:55.927210    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"]
Oct 30 18:50:55.953319 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:55.953282    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:55.953493 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:55.953324    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8882aef-7227-4781-8645-ed9c06928737-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:55.953493 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:55.953417    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:55.953493 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:55.953484    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8882aef-7227-4781-8645-ed9c06928737-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:55.953676 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:55.953502    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c9b8\" (UniqueName: \"kubernetes.io/projected/b8882aef-7227-4781-8645-ed9c06928737-kube-api-access-6c9b8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:55.953676 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:55.953606    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:55.953676 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:55.953647    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:55.953801 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:55.953699    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8882aef-7227-4781-8645-ed9c06928737-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:55.953801 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:55.953718    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.055049 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.055016    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.055268 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.055074    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8882aef-7227-4781-8645-ed9c06928737-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.055268 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.055091    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c9b8\" (UniqueName: \"kubernetes.io/projected/b8882aef-7227-4781-8645-ed9c06928737-kube-api-access-6c9b8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.055268 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.055116    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.055452 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.055287    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.055452 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.055320    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8882aef-7227-4781-8645-ed9c06928737-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.055452 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.055341    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.055452 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.055422    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.055452 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.055452    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8882aef-7227-4781-8645-ed9c06928737-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.055659 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.055502    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.055659 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.055541    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.055659 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.055579    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.055851 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.055819    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.056172 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.056124    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8882aef-7227-4781-8645-ed9c06928737-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.057761 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.057732    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8882aef-7227-4781-8645-ed9c06928737-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.058021 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.058000    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8882aef-7227-4781-8645-ed9c06928737-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.112534 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.112464    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8882aef-7227-4781-8645-ed9c06928737-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.112697 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.112601    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c9b8\" (UniqueName: \"kubernetes.io/projected/b8882aef-7227-4781-8645-ed9c06928737-kube-api-access-6c9b8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9r6z2\" (UID: \"b8882aef-7227-4781-8645-ed9c06928737\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.136461 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.136426    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:56.564800 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.564778    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"]
Oct 30 18:50:56.567685 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:50:56.567658    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8882aef_7227_4781_8645_ed9c06928737.slice/crio-f01895b1dfe39d047317453816bffa5d66458bb4210fb17da4be5772eb8f6be0 WatchSource:0}: Error finding container f01895b1dfe39d047317453816bffa5d66458bb4210fb17da4be5772eb8f6be0: Status 404 returned error can't find the container with id f01895b1dfe39d047317453816bffa5d66458bb4210fb17da4be5772eb8f6be0
Oct 30 18:50:56.569745 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.569707    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:50:56.569814 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.569794    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:50:56.569850 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.569836    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:50:56.618492 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:56.618461    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2" event={"ID":"b8882aef-7227-4781-8645-ed9c06928737","Type":"ContainerStarted","Data":"f01895b1dfe39d047317453816bffa5d66458bb4210fb17da4be5772eb8f6be0"}
Oct 30 18:50:57.341211 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.341175    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-8cxb9"]
Oct 30 18:50:57.345081 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.345064    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9"
Oct 30 18:50:57.396765 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.396741    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 18:50:57.396941 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.396924    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 18:50:57.456175 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.447584    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-8cxb9"]
Oct 30 18:50:57.473838 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.473793    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-metrics-certs\") pod \"ai-gateway-controller-84dff4756b-8cxb9\" (UID: \"7a1c50f5-54a1-47ee-99b3-2aad178e3de5\") " pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9"
Oct 30 18:50:57.474049 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.473927    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsrt\" (UniqueName: \"kubernetes.io/projected/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-kube-api-access-nhsrt\") pod \"ai-gateway-controller-84dff4756b-8cxb9\" (UID: \"7a1c50f5-54a1-47ee-99b3-2aad178e3de5\") " pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9"
Oct 30 18:50:57.497313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.497285    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-bqzpm\""
Oct 30 18:50:57.575640 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.575402    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsrt\" (UniqueName: \"kubernetes.io/projected/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-kube-api-access-nhsrt\") pod \"ai-gateway-controller-84dff4756b-8cxb9\" (UID: \"7a1c50f5-54a1-47ee-99b3-2aad178e3de5\") " pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9"
Oct 30 18:50:57.575640 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.575565    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-metrics-certs\") pod \"ai-gateway-controller-84dff4756b-8cxb9\" (UID: \"7a1c50f5-54a1-47ee-99b3-2aad178e3de5\") " pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9"
Oct 30 18:50:57.580586 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.580553    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-metrics-certs\") pod \"ai-gateway-controller-84dff4756b-8cxb9\" (UID: \"7a1c50f5-54a1-47ee-99b3-2aad178e3de5\") " pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9"
Oct 30 18:50:57.625664 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.625577    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2" event={"ID":"b8882aef-7227-4781-8645-ed9c06928737","Type":"ContainerStarted","Data":"5de6ed03f210c708d87c0f5bab65aed561185647aaca73ccce7e8a3a8185d9a9"}
Oct 30 18:50:57.634383 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.634357    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsrt\" (UniqueName: \"kubernetes.io/projected/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-kube-api-access-nhsrt\") pod \"ai-gateway-controller-84dff4756b-8cxb9\" (UID: \"7a1c50f5-54a1-47ee-99b3-2aad178e3de5\") " pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9"
Oct 30 18:50:57.683315 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.683280    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9"
Oct 30 18:50:57.741470 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:57.741410    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2" podStartSLOduration=2.74139026 podStartE2EDuration="2.74139026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:57.741366988 +0000 UTC m=+696.384534603" watchObservedRunningTime="2026-10-30 18:50:57.74139026 +0000 UTC m=+696.384557877"
Oct 30 18:50:58.067222 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:58.063695    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-8cxb9"]
Oct 30 18:50:58.075951 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:50:58.075898    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1c50f5_54a1_47ee_99b3_2aad178e3de5.slice/crio-403d52038f067716bbbb24785e9b5e0e0e959862fffe59a3c17bf1c4eadb6a50 WatchSource:0}: Error finding container 403d52038f067716bbbb24785e9b5e0e0e959862fffe59a3c17bf1c4eadb6a50: Status 404 returned error can't find the container with id 403d52038f067716bbbb24785e9b5e0e0e959862fffe59a3c17bf1c4eadb6a50
Oct 30 18:50:58.137295 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:58.137262    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:50:58.138563 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:58.138542    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9r6z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:50:58.138614 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:58.138593    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2" podUID="b8882aef-7227-4781-8645-ed9c06928737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 18:50:58.631623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:58.631592    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9" event={"ID":"7a1c50f5-54a1-47ee-99b3-2aad178e3de5","Type":"ContainerStarted","Data":"403d52038f067716bbbb24785e9b5e0e0e959862fffe59a3c17bf1c4eadb6a50"}
Oct 30 18:50:59.137418 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:59.137383    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9r6z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:50:59.137596 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:50:59.137434    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2" podUID="b8882aef-7227-4781-8645-ed9c06928737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 18:51:00.137591 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:00.137551    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9r6z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:51:00.138019 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:00.137621    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2" podUID="b8882aef-7227-4781-8645-ed9c06928737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 18:51:00.493261 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:00.493176    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7978c8786f-hn7br"]
Oct 30 18:51:00.493464 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:00.493403    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7978c8786f-hn7br" podUID="b5234541-83d3-4952-bce1-498435d99e3d" containerName="authorino" containerID="cri-o://374b14e06512932968b68f95db576b7c4b9ab422a67adeeb1e02418c8c1d0dfe" gracePeriod=30
Oct 30 18:51:00.668166 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:00.663971    6014 generic.go:356] "Generic (PLEG): container finished" podID="b5234541-83d3-4952-bce1-498435d99e3d" containerID="374b14e06512932968b68f95db576b7c4b9ab422a67adeeb1e02418c8c1d0dfe" exitCode=0
Oct 30 18:51:00.668166 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:00.664047    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7978c8786f-hn7br" event={"ID":"b5234541-83d3-4952-bce1-498435d99e3d","Type":"ContainerDied","Data":"374b14e06512932968b68f95db576b7c4b9ab422a67adeeb1e02418c8c1d0dfe"}
Oct 30 18:51:00.817412 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:00.817384    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7978c8786f-hn7br"
Oct 30 18:51:00.917518 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:00.917478    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5234541-83d3-4952-bce1-498435d99e3d-tls-cert\" (UniqueName: \"kubernetes.io/secret/b5234541-83d3-4952-bce1-498435d99e3d-tls-cert\") pod \"b5234541-83d3-4952-bce1-498435d99e3d\" (UID: \"b5234541-83d3-4952-bce1-498435d99e3d\") "
Oct 30 18:51:00.917718 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:00.917555    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5234541-83d3-4952-bce1-498435d99e3d-kube-api-access-hth56\" (UniqueName: \"kubernetes.io/projected/b5234541-83d3-4952-bce1-498435d99e3d-kube-api-access-hth56\") pod \"b5234541-83d3-4952-bce1-498435d99e3d\" (UID: \"b5234541-83d3-4952-bce1-498435d99e3d\") "
Oct 30 18:51:00.920722 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:00.920683    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5234541-83d3-4952-bce1-498435d99e3d-kube-api-access-hth56" pod "b5234541-83d3-4952-bce1-498435d99e3d" (UID: "b5234541-83d3-4952-bce1-498435d99e3d"). InnerVolumeSpecName "kube-api-access-hth56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:00.931043 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:00.930997    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5234541-83d3-4952-bce1-498435d99e3d-tls-cert" pod "b5234541-83d3-4952-bce1-498435d99e3d" (UID: "b5234541-83d3-4952-bce1-498435d99e3d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:01.019215 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:01.019178    6014 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b5234541-83d3-4952-bce1-498435d99e3d-tls-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.019215 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:01.019213    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hth56\" (UniqueName: \"kubernetes.io/projected/b5234541-83d3-4952-bce1-498435d99e3d-kube-api-access-hth56\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.137471 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:01.137430    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9r6z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:51:01.137658 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:01.137497    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2" podUID="b8882aef-7227-4781-8645-ed9c06928737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 18:51:01.677875 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:01.677844    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7978c8786f-hn7br"
Oct 30 18:51:01.678071 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:01.677843    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7978c8786f-hn7br" event={"ID":"b5234541-83d3-4952-bce1-498435d99e3d","Type":"ContainerDied","Data":"9fa96cc7d2dc02e970e170ebcc002943c8b67018f33c44d2da47f8a6cb4b9d6e"}
Oct 30 18:51:01.678071 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:01.677996    6014 scope.go:122] "RemoveContainer" containerID="374b14e06512932968b68f95db576b7c4b9ab422a67adeeb1e02418c8c1d0dfe"
Oct 30 18:51:01.839378 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:01.839345    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7978c8786f-hn7br"]
Oct 30 18:51:01.843296 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:01.843273    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7978c8786f-hn7br"]
Oct 30 18:51:02.137303 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:02.137216    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9r6z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:51:02.137303 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:02.137278    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2" podUID="b8882aef-7227-4781-8645-ed9c06928737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 18:51:03.137102 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:03.137064    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9r6z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:51:03.137528 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:03.137134    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2" podUID="b8882aef-7227-4781-8645-ed9c06928737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 18:51:03.475852 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:03.475764    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5234541-83d3-4952-bce1-498435d99e3d" path="/var/lib/kubelet/pods/b5234541-83d3-4952-bce1-498435d99e3d/volumes"
Oct 30 18:51:04.137728 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:04.137691    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9r6z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:51:04.138266 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:04.137755    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2" podUID="b8882aef-7227-4781-8645-ed9c06928737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 18:51:04.697607 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:04.697569    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9" event={"ID":"7a1c50f5-54a1-47ee-99b3-2aad178e3de5","Type":"ContainerStarted","Data":"59b22793799e4f761cf7b3d8fa5d66b25749607f1e4555dcd33a4bff8e3c5c66"}
Oct 30 18:51:04.697797 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:04.697695    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9"
Oct 30 18:51:04.812072 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:04.811942    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9" podStartSLOduration=1.42110453 podStartE2EDuration="7.811920484s" totalImagesPullingTime="6.390815954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:04.81062528 +0000 UTC m=+703.453792904" watchObservedRunningTime="2026-10-30 18:51:04.811920484 +0000 UTC m=+703.455088102"
Oct 30 18:51:05.141623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:05.141591    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:51:05.142078 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:05.141869    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:51:05.142727 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:05.142710    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2"
Oct 30 18:51:08.767615 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.767585    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm"]
Oct 30 18:51:08.768037 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.768020    6014 cpu_manager.go:414] "containerMap: removing container" podUID="b5234541-83d3-4952-bce1-498435d99e3d" containerName="authorino"
Oct 30 18:51:08.768037 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.768035    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5234541-83d3-4952-bce1-498435d99e3d" containerName="authorino"
Oct 30 18:51:08.768245 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.768165    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5234541-83d3-4952-bce1-498435d99e3d" containerName="authorino"
Oct 30 18:51:08.771341 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.771325    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm"
Oct 30 18:51:08.823201 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.823172    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rcg6p\""
Oct 30 18:51:08.872578 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.872535    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 18:51:08.874820 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.874789    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm"]
Oct 30 18:51:08.895247 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.895212    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ca6136-afbb-43d7-96ac-9b501a95c63e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-v2wnm\" (UID: \"21ca6136-afbb-43d7-96ac-9b501a95c63e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm"
Oct 30 18:51:08.895411 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.895282    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98bg9\" (UniqueName: \"kubernetes.io/projected/21ca6136-afbb-43d7-96ac-9b501a95c63e-kube-api-access-98bg9\") pod \"payload-pre-processing-7b98bb678c-v2wnm\" (UID: \"21ca6136-afbb-43d7-96ac-9b501a95c63e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm"
Oct 30 18:51:08.924377 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.924338    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-2jfvg"]
Oct 30 18:51:08.928040 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.928018    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-2jfvg"
Oct 30 18:51:08.984264 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.984233    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-2jfvg"]
Oct 30 18:51:08.996197 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.996159    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ca6136-afbb-43d7-96ac-9b501a95c63e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-v2wnm\" (UID: \"21ca6136-afbb-43d7-96ac-9b501a95c63e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm"
Oct 30 18:51:08.996404 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.996241    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bg9\" (UniqueName: \"kubernetes.io/projected/21ca6136-afbb-43d7-96ac-9b501a95c63e-kube-api-access-98bg9\") pod \"payload-pre-processing-7b98bb678c-v2wnm\" (UID: \"21ca6136-afbb-43d7-96ac-9b501a95c63e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm"
Oct 30 18:51:08.996404 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.996273    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc63475-ec73-451a-8c47-0dd82495ed48-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-2jfvg\" (UID: \"5cc63475-ec73-451a-8c47-0dd82495ed48\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2jfvg"
Oct 30 18:51:08.996404 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.996343    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlpts\" (UniqueName: \"kubernetes.io/projected/5cc63475-ec73-451a-8c47-0dd82495ed48-kube-api-access-wlpts\") pod \"payload-processing-55fbbbf9dc-2jfvg\" (UID: \"5cc63475-ec73-451a-8c47-0dd82495ed48\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2jfvg"
Oct 30 18:51:08.996933 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:08.996909    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ca6136-afbb-43d7-96ac-9b501a95c63e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-v2wnm\" (UID: \"21ca6136-afbb-43d7-96ac-9b501a95c63e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm"
Oct 30 18:51:09.052784 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:09.052712    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98bg9\" (UniqueName: \"kubernetes.io/projected/21ca6136-afbb-43d7-96ac-9b501a95c63e-kube-api-access-98bg9\") pod \"payload-pre-processing-7b98bb678c-v2wnm\" (UID: \"21ca6136-afbb-43d7-96ac-9b501a95c63e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm"
Oct 30 18:51:09.084193 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:09.084159    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm"
Oct 30 18:51:09.097861 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:09.097825    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc63475-ec73-451a-8c47-0dd82495ed48-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-2jfvg\" (UID: \"5cc63475-ec73-451a-8c47-0dd82495ed48\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2jfvg"
Oct 30 18:51:09.097981 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:09.097922    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlpts\" (UniqueName: \"kubernetes.io/projected/5cc63475-ec73-451a-8c47-0dd82495ed48-kube-api-access-wlpts\") pod \"payload-processing-55fbbbf9dc-2jfvg\" (UID: \"5cc63475-ec73-451a-8c47-0dd82495ed48\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2jfvg"
Oct 30 18:51:09.098547 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:09.098525    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc63475-ec73-451a-8c47-0dd82495ed48-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-2jfvg\" (UID: \"5cc63475-ec73-451a-8c47-0dd82495ed48\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2jfvg"
Oct 30 18:51:09.170971 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:09.170935    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlpts\" (UniqueName: \"kubernetes.io/projected/5cc63475-ec73-451a-8c47-0dd82495ed48-kube-api-access-wlpts\") pod \"payload-processing-55fbbbf9dc-2jfvg\" (UID: \"5cc63475-ec73-451a-8c47-0dd82495ed48\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2jfvg"
Oct 30 18:51:09.245441 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:09.245398    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-2jfvg"
Oct 30 18:51:09.458101 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:09.458068    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm"]
Oct 30 18:51:09.460647 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:51:09.460614    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ca6136_afbb_43d7_96ac_9b501a95c63e.slice/crio-4a4640f437aa32e95f91e6884b6432d93853eabc8e45bbed4d51dbd095695cc1 WatchSource:0}: Error finding container 4a4640f437aa32e95f91e6884b6432d93853eabc8e45bbed4d51dbd095695cc1: Status 404 returned error can't find the container with id 4a4640f437aa32e95f91e6884b6432d93853eabc8e45bbed4d51dbd095695cc1
Oct 30 18:51:09.617409 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:09.617378    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-2jfvg"]
Oct 30 18:51:09.621092 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:51:09.621056    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc63475_ec73_451a_8c47_0dd82495ed48.slice/crio-44145bf893f9574effc5ba7b21c853e6fe3e06ffb6087161e705ab5f417fb32d WatchSource:0}: Error finding container 44145bf893f9574effc5ba7b21c853e6fe3e06ffb6087161e705ab5f417fb32d: Status 404 returned error can't find the container with id 44145bf893f9574effc5ba7b21c853e6fe3e06ffb6087161e705ab5f417fb32d
Oct 30 18:51:09.729687 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:09.729574    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-2jfvg" event={"ID":"5cc63475-ec73-451a-8c47-0dd82495ed48","Type":"ContainerStarted","Data":"44145bf893f9574effc5ba7b21c853e6fe3e06ffb6087161e705ab5f417fb32d"}
Oct 30 18:51:09.730829 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:09.730797    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm" event={"ID":"21ca6136-afbb-43d7-96ac-9b501a95c63e","Type":"ContainerStarted","Data":"4a4640f437aa32e95f91e6884b6432d93853eabc8e45bbed4d51dbd095695cc1"}
Oct 30 18:51:12.749259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:12.749166    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-2jfvg" event={"ID":"5cc63475-ec73-451a-8c47-0dd82495ed48","Type":"ContainerStarted","Data":"21de99a9da190eeabefd584068536bcb724adf812ca7327b420aa018e30c1acc"}
Oct 30 18:51:12.749692 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:12.749382    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-2jfvg"
Oct 30 18:51:12.750700 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:12.750673    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm" event={"ID":"21ca6136-afbb-43d7-96ac-9b501a95c63e","Type":"ContainerStarted","Data":"01bd879c40d6e4b50e840f61b0437a8708fc833309ee68f1dc40ff8fe60a4fc4"}
Oct 30 18:51:12.750825 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:12.750797    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm"
Oct 30 18:51:12.861527 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:12.861477    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-2jfvg" podStartSLOduration=1.967092592 podStartE2EDuration="4.861464981s" totalImagesPullingTime="2.894372389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:12.860333929 +0000 UTC m=+711.503501546" watchObservedRunningTime="2026-10-30 18:51:12.861464981 +0000 UTC m=+711.504632595"
Oct 30 18:51:12.972482 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:12.972281    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm" podStartSLOduration=1.951965613 podStartE2EDuration="4.972261972s" totalImagesPullingTime="3.020296359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:12.971347167 +0000 UTC m=+711.614514811" watchObservedRunningTime="2026-10-30 18:51:12.972261972 +0000 UTC m=+711.615429587"
Oct 30 18:51:15.704797 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:15.704768    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9"
Oct 30 18:51:16.047897 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:16.047808    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2"]
Oct 30 18:51:16.052168 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:16.052125    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2"
Oct 30 18:51:16.108671 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:16.108639    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2"]
Oct 30 18:51:16.172534 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:16.172502    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54639bef-cb05-4ea2-8d95-bc17c5bb6cb9-metrics-certs\") pod \"ai-gateway-controller-58bf7d8f44-rrkp2\" (UID: \"54639bef-cb05-4ea2-8d95-bc17c5bb6cb9\") " pod="opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2"
Oct 30 18:51:16.172675 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:16.172618    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx75w\" (UniqueName: \"kubernetes.io/projected/54639bef-cb05-4ea2-8d95-bc17c5bb6cb9-kube-api-access-fx75w\") pod \"ai-gateway-controller-58bf7d8f44-rrkp2\" (UID: \"54639bef-cb05-4ea2-8d95-bc17c5bb6cb9\") " pod="opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2"
Oct 30 18:51:16.273883 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:16.273846    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx75w\" (UniqueName: \"kubernetes.io/projected/54639bef-cb05-4ea2-8d95-bc17c5bb6cb9-kube-api-access-fx75w\") pod \"ai-gateway-controller-58bf7d8f44-rrkp2\" (UID: \"54639bef-cb05-4ea2-8d95-bc17c5bb6cb9\") " pod="opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2"
Oct 30 18:51:16.274058 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:16.273943    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54639bef-cb05-4ea2-8d95-bc17c5bb6cb9-metrics-certs\") pod \"ai-gateway-controller-58bf7d8f44-rrkp2\" (UID: \"54639bef-cb05-4ea2-8d95-bc17c5bb6cb9\") " pod="opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2"
Oct 30 18:51:16.276521 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:16.276503    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54639bef-cb05-4ea2-8d95-bc17c5bb6cb9-metrics-certs\") pod \"ai-gateway-controller-58bf7d8f44-rrkp2\" (UID: \"54639bef-cb05-4ea2-8d95-bc17c5bb6cb9\") " pod="opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2"
Oct 30 18:51:16.330790 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:16.330752    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx75w\" (UniqueName: \"kubernetes.io/projected/54639bef-cb05-4ea2-8d95-bc17c5bb6cb9-kube-api-access-fx75w\") pod \"ai-gateway-controller-58bf7d8f44-rrkp2\" (UID: \"54639bef-cb05-4ea2-8d95-bc17c5bb6cb9\") " pod="opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2"
Oct 30 18:51:16.364708 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:16.364668    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2"
Oct 30 18:51:16.739188 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:16.739162    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2"]
Oct 30 18:51:16.741796 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:51:16.741765    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54639bef_cb05_4ea2_8d95_bc17c5bb6cb9.slice/crio-4dc30e185efd4fd9d23bace7b4e0dc5d3984b5ba303f4618fca41b75c04b35b1 WatchSource:0}: Error finding container 4dc30e185efd4fd9d23bace7b4e0dc5d3984b5ba303f4618fca41b75c04b35b1: Status 404 returned error can't find the container with id 4dc30e185efd4fd9d23bace7b4e0dc5d3984b5ba303f4618fca41b75c04b35b1
Oct 30 18:51:16.773185 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:16.773152    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2" event={"ID":"54639bef-cb05-4ea2-8d95-bc17c5bb6cb9","Type":"ContainerStarted","Data":"4dc30e185efd4fd9d23bace7b4e0dc5d3984b5ba303f4618fca41b75c04b35b1"}
Oct 30 18:51:17.778964 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:17.778930    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2" event={"ID":"54639bef-cb05-4ea2-8d95-bc17c5bb6cb9","Type":"ContainerStarted","Data":"2a7e053ffb6f147049e9afd7f1296e36c28d902475ca30cb587b73dd2f0f9cc4"}
Oct 30 18:51:17.779420 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:17.779056    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2"
Oct 30 18:51:17.893773 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:17.893643    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2" podStartSLOduration=1.02773927 podStartE2EDuration="1.893525884s" totalImagesPullingTime="865.786614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:17.892188131 +0000 UTC m=+716.535355742" watchObservedRunningTime="2026-10-30 18:51:17.893525884 +0000 UTC m=+716.536693499"
Oct 30 18:51:23.257329 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:51:23.257291    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:51:23.759672 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:23.759644    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-2jfvg"
Oct 30 18:51:23.759862 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:23.759707    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v2wnm"
Oct 30 18:51:28.786154 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:28.786116    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-58bf7d8f44-rrkp2"
Oct 30 18:51:28.950528 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:28.950498    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-8cxb9"]
Oct 30 18:51:28.950741 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:28.950720    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9" podUID="7a1c50f5-54a1-47ee-99b3-2aad178e3de5" containerName="manager" containerID="cri-o://59b22793799e4f761cf7b3d8fa5d66b25749607f1e4555dcd33a4bff8e3c5c66" gracePeriod=30
Oct 30 18:51:29.201916 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.201883    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9"
Oct 30 18:51:29.303516 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.303472    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-kube-api-access-nhsrt\" (UniqueName: \"kubernetes.io/projected/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-kube-api-access-nhsrt\") pod \"7a1c50f5-54a1-47ee-99b3-2aad178e3de5\" (UID: \"7a1c50f5-54a1-47ee-99b3-2aad178e3de5\") "
Oct 30 18:51:29.303694 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.303628    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-metrics-certs\") pod \"7a1c50f5-54a1-47ee-99b3-2aad178e3de5\" (UID: \"7a1c50f5-54a1-47ee-99b3-2aad178e3de5\") "
Oct 30 18:51:29.305884 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.305859    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-metrics-certs" pod "7a1c50f5-54a1-47ee-99b3-2aad178e3de5" (UID: "7a1c50f5-54a1-47ee-99b3-2aad178e3de5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:29.305971 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.305884    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-kube-api-access-nhsrt" pod "7a1c50f5-54a1-47ee-99b3-2aad178e3de5" (UID: "7a1c50f5-54a1-47ee-99b3-2aad178e3de5"). InnerVolumeSpecName "kube-api-access-nhsrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:29.405178 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.405066    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:51:29.405178 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.405095    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhsrt\" (UniqueName: \"kubernetes.io/projected/7a1c50f5-54a1-47ee-99b3-2aad178e3de5-kube-api-access-nhsrt\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:51:29.840895 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.840861    6014 generic.go:356] "Generic (PLEG): container finished" podID="7a1c50f5-54a1-47ee-99b3-2aad178e3de5" containerID="59b22793799e4f761cf7b3d8fa5d66b25749607f1e4555dcd33a4bff8e3c5c66" exitCode=0
Oct 30 18:51:29.841313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.840918    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9"
Oct 30 18:51:29.841313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.840947    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9" event={"ID":"7a1c50f5-54a1-47ee-99b3-2aad178e3de5","Type":"ContainerDied","Data":"59b22793799e4f761cf7b3d8fa5d66b25749607f1e4555dcd33a4bff8e3c5c66"}
Oct 30 18:51:29.841313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.841028    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-8cxb9" event={"ID":"7a1c50f5-54a1-47ee-99b3-2aad178e3de5","Type":"ContainerDied","Data":"403d52038f067716bbbb24785e9b5e0e0e959862fffe59a3c17bf1c4eadb6a50"}
Oct 30 18:51:29.841313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.841059    6014 scope.go:122] "RemoveContainer" containerID="59b22793799e4f761cf7b3d8fa5d66b25749607f1e4555dcd33a4bff8e3c5c66"
Oct 30 18:51:29.850065 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.850045    6014 scope.go:122] "RemoveContainer" containerID="59b22793799e4f761cf7b3d8fa5d66b25749607f1e4555dcd33a4bff8e3c5c66"
Oct 30 18:51:29.850372 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:51:29.850343    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b22793799e4f761cf7b3d8fa5d66b25749607f1e4555dcd33a4bff8e3c5c66\": container with ID starting with 59b22793799e4f761cf7b3d8fa5d66b25749607f1e4555dcd33a4bff8e3c5c66 not found: ID does not exist" containerID="59b22793799e4f761cf7b3d8fa5d66b25749607f1e4555dcd33a4bff8e3c5c66"
Oct 30 18:51:29.850443 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:29.850382    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b22793799e4f761cf7b3d8fa5d66b25749607f1e4555dcd33a4bff8e3c5c66"} err="failed to get container status \"59b22793799e4f761cf7b3d8fa5d66b25749607f1e4555dcd33a4bff8e3c5c66\": rpc error: code = NotFound desc = could not find container \"59b22793799e4f761cf7b3d8fa5d66b25749607f1e4555dcd33a4bff8e3c5c66\": container with ID starting with 59b22793799e4f761cf7b3d8fa5d66b25749607f1e4555dcd33a4bff8e3c5c66 not found: ID does not exist"
Oct 30 18:51:30.004456 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:30.004425    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-8cxb9"]
Oct 30 18:51:30.053682 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:30.053648    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-8cxb9"]
Oct 30 18:51:31.474681 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:31.474609    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a1c50f5-54a1-47ee-99b3-2aad178e3de5" path="/var/lib/kubelet/pods/7a1c50f5-54a1-47ee-99b3-2aad178e3de5/volumes"
Oct 30 18:51:53.256538 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:51:53.256494    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:51:59.719280 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.719242    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"]
Oct 30 18:51:59.719918 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.719898    6014 cpu_manager.go:414] "containerMap: removing container" podUID="7a1c50f5-54a1-47ee-99b3-2aad178e3de5" containerName="manager"
Oct 30 18:51:59.720004 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.719921    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1c50f5-54a1-47ee-99b3-2aad178e3de5" containerName="manager"
Oct 30 18:51:59.720075 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.720062    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a1c50f5-54a1-47ee-99b3-2aad178e3de5" containerName="manager"
Oct 30 18:51:59.728766 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.728743    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.802043 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.802000    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"]
Oct 30 18:51:59.822363 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.822325    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.822546 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.822402    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.822546 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.822471    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.822546 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.822503    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4q59\" (UniqueName: \"kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-kube-api-access-l4q59\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.822546 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.822536    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.822713 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.822647    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.822713 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.822666    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.822713 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.822688    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.822713 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.822712    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.830133 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.830112    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-41db7c-openshift-default-dockercfg-fhcxz\""
Oct 30 18:51:59.924271 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.924238    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.924464 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.924302    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.924464 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.924326    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.924464 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.924359    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.924464 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.924387    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.924464 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.924455    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.924742 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.924532    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.924742 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.924565    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.924742 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.924594    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4q59\" (UniqueName: \"kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-kube-api-access-l4q59\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.924742 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.924662    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.924948 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.924780    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.924948 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.924928    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.925031 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.924995    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.925527 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.925498    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.926819 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.926790    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.926973 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.926948    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.981045 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.980973    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:51:59.981226 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:51:59.981059    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4q59\" (UniqueName: \"kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-kube-api-access-l4q59\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:52:00.042124 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.042088    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:52:00.048523 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.048496    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"]
Oct 30 18:52:00.053755 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.053733    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.106933 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.106901    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f5763a-openshift-default-dockercfg-hrzwg\""
Oct 30 18:52:00.117621 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.117589    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"]
Oct 30 18:52:00.228045 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.228011    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.228239 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.228054    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.228239 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.228072    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.228239 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.228154    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.228347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.228268    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.228347 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.228321    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.228410 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.228354    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.228445 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.228417    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl8fs\" (UniqueName: \"kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-kube-api-access-wl8fs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.228445 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.228441    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.329490 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.329458    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.329667 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.329501    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.329667 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.329543    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.329667 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.329586    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8fs\" (UniqueName: \"kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-kube-api-access-wl8fs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.329857 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.329718    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.329857 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.329830    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.329958 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.329871    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.329958 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.329899    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.329958 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.329907    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.329958 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.329928    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.329958 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.329944    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.330241 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.330002    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.330241 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.330025    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.330483 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.330465    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.332172 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.332151    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.332480 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.332462    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.386245 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.386212    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.386513 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.386497    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8fs\" (UniqueName: \"kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-kube-api-access-wl8fs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.491953 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.491928    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"]
Oct 30 18:52:00.494512 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:52:00.494476    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13d3d6b_d787_4b0e_a46a_4a132174f16f.slice/crio-1db1f7dae32cbd7784dc650213fcd280cadaf321dcd696e067f99f6ae7dd6d9e WatchSource:0}: Error finding container 1db1f7dae32cbd7784dc650213fcd280cadaf321dcd696e067f99f6ae7dd6d9e: Status 404 returned error can't find the container with id 1db1f7dae32cbd7784dc650213fcd280cadaf321dcd696e067f99f6ae7dd6d9e
Oct 30 18:52:00.496843 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.496808    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:00.496919 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.496894    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:00.496955 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.496932    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:00.677089 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.677002    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:00.683321 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.683294    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"]
Oct 30 18:52:00.687895 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.687860    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.740101 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.740075    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-975cf5-openshift-default-dockercfg-g97ps\""
Oct 30 18:52:00.749443 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.748530    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"]
Oct 30 18:52:00.836800 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.836768    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.836988 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.836820    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.836988 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.836912    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.837117 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.837003    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s99v8\" (UniqueName: \"kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-kube-api-access-s99v8\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.837476 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.837152    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.837476 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.837205    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.837476 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.837276    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.837476 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.837305    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.837476 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.837322    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.938248 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.938127    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s99v8\" (UniqueName: \"kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-kube-api-access-s99v8\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.938421 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.938254    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.938421 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.938289    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.938421 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.938347    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.938421 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.938372    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.938421 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.938397    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.938689 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.938464    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.938689 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.938494    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.938689 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.938537    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.938978 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.938957    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.939089 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.939046    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.939573 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.939375    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.940325 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.940307    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.940585 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.940527    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.941715 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.941695    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.942024 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.941970    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.987928 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.987894    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"]
Oct 30 18:52:00.992849 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.992822    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:00.996313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.996284    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" event={"ID":"a13d3d6b-d787-4b0e-a46a-4a132174f16f","Type":"ContainerStarted","Data":"4ccad808aac5beeb6f8c43f2aa9661459394e46bb09aa63970586304dd693ab7"}
Oct 30 18:52:00.996449 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.996333    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" event={"ID":"a13d3d6b-d787-4b0e-a46a-4a132174f16f","Type":"ContainerStarted","Data":"1db1f7dae32cbd7784dc650213fcd280cadaf321dcd696e067f99f6ae7dd6d9e"}
Oct 30 18:52:00.996648 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.996621    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:00.996798 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:00.996779    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s99v8\" (UniqueName: \"kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-kube-api-access-s99v8\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:01.001728 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.001709    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:01.043238 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.043206    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:52:01.044499 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.044471    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-611053-openshift-default-dockercfg-7h2nn\""
Oct 30 18:52:01.049701 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.049678    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:52:01.056031 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.056007    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"]
Oct 30 18:52:01.065228 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.062825    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"]
Oct 30 18:52:01.068297 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:52:01.068267    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9eb37d_8fc9_4b7d_90f9_6792ecbd7e84.slice/crio-a40b3f279c8c5685a81fecd2d4d5e76abc3d1a1554a25d08636660e776f23e93 WatchSource:0}: Error finding container a40b3f279c8c5685a81fecd2d4d5e76abc3d1a1554a25d08636660e776f23e93: Status 404 returned error can't find the container with id a40b3f279c8c5685a81fecd2d4d5e76abc3d1a1554a25d08636660e776f23e93
Oct 30 18:52:01.076762 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.076717    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:01.076880 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.076817    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:01.076880 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.076863    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:01.147130 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.145835    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.147130 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.145959    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945375bc-250a-4b52-8cfc-e405096baf26-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.147130 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.146011    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.147130 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.146071    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945375bc-250a-4b52-8cfc-e405096baf26-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.147130 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.146121    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.147130 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.146184    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.147130 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.146225    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.147130 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.146271    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwp5v\" (UniqueName: \"kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-kube-api-access-vwp5v\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.147130 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.146311    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.227535 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.227420    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" podStartSLOduration=2.227404214 podStartE2EDuration="2.227404214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:01.225704319 +0000 UTC m=+759.868871943" watchObservedRunningTime="2026-10-30 18:52:01.227404214 +0000 UTC m=+759.870571899"
Oct 30 18:52:01.247228 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.247194    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945375bc-250a-4b52-8cfc-e405096baf26-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.247394 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.247250    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.247394 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.247288    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945375bc-250a-4b52-8cfc-e405096baf26-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.247394 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.247313    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.247394 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.247356    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.247394 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.247389    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.247661 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.247415    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwp5v\" (UniqueName: \"kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-kube-api-access-vwp5v\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.247661 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.247438    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.247661 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.247490    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.247816 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.247799    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.247913 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.247884    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.248044 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.247947    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.248044 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.247954    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.248124 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.248085    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945375bc-250a-4b52-8cfc-e405096baf26-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.249866 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.249844    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.249957 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.249943    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945375bc-250a-4b52-8cfc-e405096baf26-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.310441 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.310409    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwp5v\" (UniqueName: \"kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-kube-api-access-vwp5v\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.312602 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.312575    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.338728 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.338697    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:01.396522 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.396490    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"]
Oct 30 18:52:01.401914 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.401370    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:01.401914 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.401453    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:01.401914 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.401499    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:01.729488 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.729456    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"]
Oct 30 18:52:01.733530 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:52:01.733495    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945375bc_250a_4b52_8cfc_e405096baf26.slice/crio-aab3ffbbb99437e8d53ed1c15be1f645d7a6fd3c5f492da531978b4d378079c8 WatchSource:0}: Error finding container aab3ffbbb99437e8d53ed1c15be1f645d7a6fd3c5f492da531978b4d378079c8: Status 404 returned error can't find the container with id aab3ffbbb99437e8d53ed1c15be1f645d7a6fd3c5f492da531978b4d378079c8
Oct 30 18:52:01.736016 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.735980    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:01.736089 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.736053    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:01.736089 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:01.736079    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:02.006368 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.006270    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" event={"ID":"cab5db1f-c822-4fe8-a692-6e8aaa8306bb","Type":"ContainerStarted","Data":"01c0757f95ab0889c11dbb6afdaef8757d59c628a0b07e3c719bad68138d0dba"}
Oct 30 18:52:02.006368 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.006348    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" event={"ID":"cab5db1f-c822-4fe8-a692-6e8aaa8306bb","Type":"ContainerStarted","Data":"72e753ccc1255e1214824aa4622a2ddf4076607b18e487912608c481b5268dcb"}
Oct 30 18:52:02.008589 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.008550    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" event={"ID":"945375bc-250a-4b52-8cfc-e405096baf26","Type":"ContainerStarted","Data":"a42f5e791085fd50c3401e21930ab7422508d5b44609144db981f97a1363bd7e"}
Oct 30 18:52:02.008692 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.008609    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" event={"ID":"945375bc-250a-4b52-8cfc-e405096baf26","Type":"ContainerStarted","Data":"aab3ffbbb99437e8d53ed1c15be1f645d7a6fd3c5f492da531978b4d378079c8"}
Oct 30 18:52:02.010191 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.010170    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4" event={"ID":"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84","Type":"ContainerStarted","Data":"e5266cc0ab19505eff5c3e1dc325c6fc1b59970fac3e54aa6c1aa2fb765e62df"}
Oct 30 18:52:02.010299 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.010215    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4" event={"ID":"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84","Type":"ContainerStarted","Data":"a40b3f279c8c5685a81fecd2d4d5e76abc3d1a1554a25d08636660e776f23e93"}
Oct 30 18:52:02.010457 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.010424    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:52:02.011375 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.011354    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:52:02.122465 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.122405    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" podStartSLOduration=2.1223883199999998 podStartE2EDuration="2.12238832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:02.122050728 +0000 UTC m=+760.765218343" watchObservedRunningTime="2026-10-30 18:52:02.12238832 +0000 UTC m=+760.765555933"
Oct 30 18:52:02.243711 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.243216    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" podStartSLOduration=2.243193867 podStartE2EDuration="2.243193867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:02.238053035 +0000 UTC m=+760.881220650" watchObservedRunningTime="2026-10-30 18:52:02.243193867 +0000 UTC m=+760.886361479"
Oct 30 18:52:02.339226 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.339190    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:02.344480 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.344454    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:02.366179 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.361929    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4" podStartSLOduration=2.36190653 podStartE2EDuration="2.36190653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:02.353846256 +0000 UTC m=+760.997013928" watchObservedRunningTime="2026-10-30 18:52:02.36190653 +0000 UTC m=+761.005074146"
Oct 30 18:52:02.678058 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.677960    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:02.683622 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:02.683596    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:03.001946 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:03.001859    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:03.006755 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:03.006733    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:03.016113 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:03.015985    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:03.016113 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:03.016036    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:03.016113 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:03.016064    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:03.016641 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:03.016620    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:52:03.016737 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:03.016671    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:52:03.016809 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:03.016800    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:08.667967 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.667921    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"]
Oct 30 18:52:08.671702 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.671685    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"
Oct 30 18:52:08.723648 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.723012    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-1290a6\""
Oct 30 18:52:08.732091 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.732053    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68730cc9-c11b-4699-a58e-792d376fe6d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6\" (UID: \"68730cc9-c11b-4699-a58e-792d376fe6d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"
Oct 30 18:52:08.732310 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.732119    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq9vf\" (UniqueName: \"kubernetes.io/projected/68730cc9-c11b-4699-a58e-792d376fe6d0-kube-api-access-sq9vf\") pod \"payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6\" (UID: \"68730cc9-c11b-4699-a58e-792d376fe6d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"
Oct 30 18:52:08.774344 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.774311    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"]
Oct 30 18:52:08.796252 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.796218    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"]
Oct 30 18:52:08.800034 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.800014    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"
Oct 30 18:52:08.832272 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.832216    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-1290a6-dockercfg-6j762\""
Oct 30 18:52:08.833679 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.833654    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c98a0a-4137-408a-9811-3a27f4866a84-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6\" (UID: \"f3c98a0a-4137-408a-9811-3a27f4866a84\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"
Oct 30 18:52:08.833749 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.833697    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68730cc9-c11b-4699-a58e-792d376fe6d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6\" (UID: \"68730cc9-c11b-4699-a58e-792d376fe6d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"
Oct 30 18:52:08.833749 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.833725    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6lph\" (UniqueName: \"kubernetes.io/projected/f3c98a0a-4137-408a-9811-3a27f4866a84-kube-api-access-w6lph\") pod \"payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6\" (UID: \"f3c98a0a-4137-408a-9811-3a27f4866a84\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"
Oct 30 18:52:08.833868 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.833840    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq9vf\" (UniqueName: \"kubernetes.io/projected/68730cc9-c11b-4699-a58e-792d376fe6d0-kube-api-access-sq9vf\") pod \"payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6\" (UID: \"68730cc9-c11b-4699-a58e-792d376fe6d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"
Oct 30 18:52:08.834842 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.834818    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68730cc9-c11b-4699-a58e-792d376fe6d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6\" (UID: \"68730cc9-c11b-4699-a58e-792d376fe6d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"
Oct 30 18:52:08.856463 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.856433    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"]
Oct 30 18:52:08.894267 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.894241    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq9vf\" (UniqueName: \"kubernetes.io/projected/68730cc9-c11b-4699-a58e-792d376fe6d0-kube-api-access-sq9vf\") pod \"payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6\" (UID: \"68730cc9-c11b-4699-a58e-792d376fe6d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"
Oct 30 18:52:08.935232 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.935111    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c98a0a-4137-408a-9811-3a27f4866a84-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6\" (UID: \"f3c98a0a-4137-408a-9811-3a27f4866a84\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"
Oct 30 18:52:08.935232 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.935197    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6lph\" (UniqueName: \"kubernetes.io/projected/f3c98a0a-4137-408a-9811-3a27f4866a84-kube-api-access-w6lph\") pod \"payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6\" (UID: \"f3c98a0a-4137-408a-9811-3a27f4866a84\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"
Oct 30 18:52:08.935912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.935889    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c98a0a-4137-408a-9811-3a27f4866a84-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6\" (UID: \"f3c98a0a-4137-408a-9811-3a27f4866a84\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"
Oct 30 18:52:08.986008 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.985972    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"
Oct 30 18:52:08.991954 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:08.991928    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6lph\" (UniqueName: \"kubernetes.io/projected/f3c98a0a-4137-408a-9811-3a27f4866a84-kube-api-access-w6lph\") pod \"payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6\" (UID: \"f3c98a0a-4137-408a-9811-3a27f4866a84\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"
Oct 30 18:52:09.113753 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:09.113722    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"
Oct 30 18:52:09.359386 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:09.359361    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"]
Oct 30 18:52:09.361806 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:52:09.361774    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68730cc9_c11b_4699_a58e_792d376fe6d0.slice/crio-83d202ffb0a90c1d771218060668b333de05f1a3919aea922118738ecb880aaf WatchSource:0}: Error finding container 83d202ffb0a90c1d771218060668b333de05f1a3919aea922118738ecb880aaf: Status 404 returned error can't find the container with id 83d202ffb0a90c1d771218060668b333de05f1a3919aea922118738ecb880aaf
Oct 30 18:52:09.492355 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:09.492322    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"]
Oct 30 18:52:09.495547 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:52:09.495516    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c98a0a_4137_408a_9811_3a27f4866a84.slice/crio-103a225b51f6431cc5bd86195263de5a336cc7dc051ba43ccc88edff351af7ad WatchSource:0}: Error finding container 103a225b51f6431cc5bd86195263de5a336cc7dc051ba43ccc88edff351af7ad: Status 404 returned error can't find the container with id 103a225b51f6431cc5bd86195263de5a336cc7dc051ba43ccc88edff351af7ad
Oct 30 18:52:10.071765 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:10.071727    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6" event={"ID":"68730cc9-c11b-4699-a58e-792d376fe6d0","Type":"ContainerStarted","Data":"e588ade265079af3558ec4a0d8892e30c59d618bb09992ed03048bbbb27b4083"}
Oct 30 18:52:10.072281 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:10.071792    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6" event={"ID":"68730cc9-c11b-4699-a58e-792d376fe6d0","Type":"ContainerStarted","Data":"83d202ffb0a90c1d771218060668b333de05f1a3919aea922118738ecb880aaf"}
Oct 30 18:52:10.072281 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:10.071941    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"
Oct 30 18:52:10.073471 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:10.073441    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6" event={"ID":"f3c98a0a-4137-408a-9811-3a27f4866a84","Type":"ContainerStarted","Data":"42493e5fe9cb498d1288b2b6ee0ff317f72aab8d8cb56627f1102358b0fcafef"}
Oct 30 18:52:10.073620 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:10.073493    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"
Oct 30 18:52:10.073620 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:10.073515    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6" event={"ID":"f3c98a0a-4137-408a-9811-3a27f4866a84","Type":"ContainerStarted","Data":"103a225b51f6431cc5bd86195263de5a336cc7dc051ba43ccc88edff351af7ad"}
Oct 30 18:52:10.186784 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:10.186722    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6" podStartSLOduration=2.18670769 podStartE2EDuration="2.18670769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:10.185548796 +0000 UTC m=+768.828716411" watchObservedRunningTime="2026-10-30 18:52:10.18670769 +0000 UTC m=+768.829875303"
Oct 30 18:52:10.304573 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:10.304409    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6" podStartSLOduration=2.304385932 podStartE2EDuration="2.304385932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:10.302255735 +0000 UTC m=+768.945423350" watchObservedRunningTime="2026-10-30 18:52:10.304385932 +0000 UTC m=+768.947553548"
Oct 30 18:52:13.501989 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:13.501961    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"]
Oct 30 18:52:13.505929 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:13.505908    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"
Oct 30 18:52:13.574163 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:13.567819    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"]
Oct 30 18:52:13.587693 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:13.587651    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb\" (UID: \"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"
Oct 30 18:52:13.588088 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:13.588065    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54mpn\" (UniqueName: \"kubernetes.io/projected/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-kube-api-access-54mpn\") pod \"payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb\" (UID: \"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"
Oct 30 18:52:13.609957 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:13.609917    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-41db7c-dockercfg-6j457\""
Oct 30 18:52:13.610176 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:13.610006    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-41db7c\""
Oct 30 18:52:13.689609 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:13.689575    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb\" (UID: \"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"
Oct 30 18:52:13.689794 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:13.689701    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54mpn\" (UniqueName: \"kubernetes.io/projected/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-kube-api-access-54mpn\") pod \"payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb\" (UID: \"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"
Oct 30 18:52:13.690318 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:13.690300    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb\" (UID: \"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"
Oct 30 18:52:13.746957 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:13.746925    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54mpn\" (UniqueName: \"kubernetes.io/projected/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-kube-api-access-54mpn\") pod \"payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb\" (UID: \"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"
Oct 30 18:52:13.819551 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:13.819514    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"
Oct 30 18:52:14.191266 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:14.191242    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"]
Oct 30 18:52:14.193266 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:52:14.193234    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd18ae9_9e76_42e1_bde3_da20fcc21ad8.slice/crio-2a575c795b7ee716f20d69997e6153af0876d5f0f1c890eb8b2de5dd1584b196 WatchSource:0}: Error finding container 2a575c795b7ee716f20d69997e6153af0876d5f0f1c890eb8b2de5dd1584b196: Status 404 returned error can't find the container with id 2a575c795b7ee716f20d69997e6153af0876d5f0f1c890eb8b2de5dd1584b196
Oct 30 18:52:15.104176 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:15.104120    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb" event={"ID":"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8","Type":"ContainerStarted","Data":"674d30df0aa1a1d439ddee5f2bd20898e0bff436d87d288357e7093515a1f121"}
Oct 30 18:52:15.104646 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:15.104235    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb" event={"ID":"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8","Type":"ContainerStarted","Data":"2a575c795b7ee716f20d69997e6153af0876d5f0f1c890eb8b2de5dd1584b196"}
Oct 30 18:52:15.104646 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:15.104313    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"
Oct 30 18:52:15.222037 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:15.221984    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb" podStartSLOduration=2.221969791 podStartE2EDuration="2.221969791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:15.219804972 +0000 UTC m=+773.862972586" watchObservedRunningTime="2026-10-30 18:52:15.221969791 +0000 UTC m=+773.865137396"
Oct 30 18:52:16.128792 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:16.128757    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"]
Oct 30 18:52:16.132919 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:16.132898    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"
Oct 30 18:52:16.188098 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:16.188065    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-66bdc0-dockercfg-8hnmc\""
Oct 30 18:52:16.226662 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:16.226611    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvkdb\" (UniqueName: \"kubernetes.io/projected/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-kube-api-access-hvkdb\") pod \"payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t\" (UID: \"ec095b92-3649-4328-b5c7-fb47d3d5a9ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"
Oct 30 18:52:16.226846 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:16.226758    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t\" (UID: \"ec095b92-3649-4328-b5c7-fb47d3d5a9ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"
Oct 30 18:52:16.237110 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:16.237081    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-66bdc0\""
Oct 30 18:52:16.239194 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:16.239173    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"]
Oct 30 18:52:16.328313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:16.328275    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvkdb\" (UniqueName: \"kubernetes.io/projected/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-kube-api-access-hvkdb\") pod \"payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t\" (UID: \"ec095b92-3649-4328-b5c7-fb47d3d5a9ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"
Oct 30 18:52:16.328509 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:16.328355    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t\" (UID: \"ec095b92-3649-4328-b5c7-fb47d3d5a9ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"
Oct 30 18:52:16.329021 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:16.329000    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t\" (UID: \"ec095b92-3649-4328-b5c7-fb47d3d5a9ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"
Oct 30 18:52:16.386713 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:16.386634    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvkdb\" (UniqueName: \"kubernetes.io/projected/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-kube-api-access-hvkdb\") pod \"payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t\" (UID: \"ec095b92-3649-4328-b5c7-fb47d3d5a9ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"
Oct 30 18:52:16.447200 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:16.447163    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"
Oct 30 18:52:16.822035 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:16.822008    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"]
Oct 30 18:52:16.824340 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:52:16.824313    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec095b92_3649_4328_b5c7_fb47d3d5a9ae.slice/crio-7ac0f95d713aeb16d0d925fff6c0402f0b23c331f1d3cc066b394b113101dab4 WatchSource:0}: Error finding container 7ac0f95d713aeb16d0d925fff6c0402f0b23c331f1d3cc066b394b113101dab4: Status 404 returned error can't find the container with id 7ac0f95d713aeb16d0d925fff6c0402f0b23c331f1d3cc066b394b113101dab4
Oct 30 18:52:17.117053 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:17.116959    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t" event={"ID":"ec095b92-3649-4328-b5c7-fb47d3d5a9ae","Type":"ContainerStarted","Data":"85bab69eb18b667a328cced4bc5f55092f03a5f8923e09a13cc635e65e455700"}
Oct 30 18:52:17.117053 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:17.117024    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"
Oct 30 18:52:17.117053 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:17.117049    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t" event={"ID":"ec095b92-3649-4328-b5c7-fb47d3d5a9ae","Type":"ContainerStarted","Data":"7ac0f95d713aeb16d0d925fff6c0402f0b23c331f1d3cc066b394b113101dab4"}
Oct 30 18:52:17.232707 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:17.232653    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t" podStartSLOduration=1.232636816 podStartE2EDuration="1.232636816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:17.230825692 +0000 UTC m=+775.873993306" watchObservedRunningTime="2026-10-30 18:52:17.232636816 +0000 UTC m=+775.875804448"
Oct 30 18:52:17.944833 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:17.944793    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:52:20.911954 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:20.911921    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"]
Oct 30 18:52:20.917522 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:20.917492    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"
Oct 30 18:52:20.973169 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:20.971259    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"]
Oct 30 18:52:21.002343 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:21.002310    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz95x\" (UniqueName: \"kubernetes.io/projected/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-kube-api-access-lz95x\") pod \"payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9\" (UID: \"03e372cb-2b3a-48a7-b6ec-fca4e17652a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"
Oct 30 18:52:21.002522 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:21.002447    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9\" (UID: \"03e372cb-2b3a-48a7-b6ec-fca4e17652a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"
Oct 30 18:52:21.018021 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:21.017999    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-975cf5-dockercfg-s4694\""
Oct 30 18:52:21.071164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:21.070318    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-975cf5\""
Oct 30 18:52:21.082873 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:21.082842    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"
Oct 30 18:52:21.083009 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:21.082938    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"
Oct 30 18:52:21.104216 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:21.104179    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz95x\" (UniqueName: \"kubernetes.io/projected/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-kube-api-access-lz95x\") pod \"payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9\" (UID: \"03e372cb-2b3a-48a7-b6ec-fca4e17652a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"
Oct 30 18:52:21.104352 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:21.104272    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9\" (UID: \"03e372cb-2b3a-48a7-b6ec-fca4e17652a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"
Oct 30 18:52:21.104903 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:21.104886    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9\" (UID: \"03e372cb-2b3a-48a7-b6ec-fca4e17652a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"
Oct 30 18:52:21.173225 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:21.173099    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz95x\" (UniqueName: \"kubernetes.io/projected/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-kube-api-access-lz95x\") pod \"payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9\" (UID: \"03e372cb-2b3a-48a7-b6ec-fca4e17652a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"
Oct 30 18:52:21.231018 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:21.230987    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"
Oct 30 18:52:21.639905 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:21.639883    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"]
Oct 30 18:52:21.642272 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:52:21.642244    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e372cb_2b3a_48a7_b6ec_fca4e17652a9.slice/crio-ff229af754b2b9fd58663a4303a36fe262789b85f0b28d1b7da69ecc3641ea88 WatchSource:0}: Error finding container ff229af754b2b9fd58663a4303a36fe262789b85f0b28d1b7da69ecc3641ea88: Status 404 returned error can't find the container with id ff229af754b2b9fd58663a4303a36fe262789b85f0b28d1b7da69ecc3641ea88
Oct 30 18:52:22.174632 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:22.174592    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9" event={"ID":"03e372cb-2b3a-48a7-b6ec-fca4e17652a9","Type":"ContainerStarted","Data":"2c4bd3b3fb12e335f7e7e3370454276118a7ee11def3634dde8a725fd292e88d"}
Oct 30 18:52:22.175051 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:22.174659    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9" event={"ID":"03e372cb-2b3a-48a7-b6ec-fca4e17652a9","Type":"ContainerStarted","Data":"ff229af754b2b9fd58663a4303a36fe262789b85f0b28d1b7da69ecc3641ea88"}
Oct 30 18:52:22.175051 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:22.174747    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"
Oct 30 18:52:22.288841 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:22.288777    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9" podStartSLOduration=2.288763835 podStartE2EDuration="2.288763835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:22.287723025 +0000 UTC m=+780.930890642" watchObservedRunningTime="2026-10-30 18:52:22.288763835 +0000 UTC m=+780.931931449"
Oct 30 18:52:23.279358 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:52:23.279328    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:52:23.291046 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:23.291018    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"]
Oct 30 18:52:23.294585 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:23.294569    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"
Oct 30 18:52:23.380081 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:23.380042    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"]
Oct 30 18:52:23.420921 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:23.420893    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-611053\""
Oct 30 18:52:23.421073 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:23.420959    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-611053-dockercfg-6jr7j\""
Oct 30 18:52:23.432700 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:23.432652    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgmxj\" (UniqueName: \"kubernetes.io/projected/7c1d3787-e640-4027-b267-88a3c8a19873-kube-api-access-xgmxj\") pod \"payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs\" (UID: \"7c1d3787-e640-4027-b267-88a3c8a19873\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"
Oct 30 18:52:23.432843 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:23.432782    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1d3787-e640-4027-b267-88a3c8a19873-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs\" (UID: \"7c1d3787-e640-4027-b267-88a3c8a19873\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"
Oct 30 18:52:23.533928 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:23.533844    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgmxj\" (UniqueName: \"kubernetes.io/projected/7c1d3787-e640-4027-b267-88a3c8a19873-kube-api-access-xgmxj\") pod \"payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs\" (UID: \"7c1d3787-e640-4027-b267-88a3c8a19873\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"
Oct 30 18:52:23.533928 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:23.533910    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1d3787-e640-4027-b267-88a3c8a19873-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs\" (UID: \"7c1d3787-e640-4027-b267-88a3c8a19873\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"
Oct 30 18:52:23.534627 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:23.534608    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1d3787-e640-4027-b267-88a3c8a19873-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs\" (UID: \"7c1d3787-e640-4027-b267-88a3c8a19873\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"
Oct 30 18:52:23.590217 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:23.590190    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgmxj\" (UniqueName: \"kubernetes.io/projected/7c1d3787-e640-4027-b267-88a3c8a19873-kube-api-access-xgmxj\") pod \"payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs\" (UID: \"7c1d3787-e640-4027-b267-88a3c8a19873\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"
Oct 30 18:52:23.608467 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:23.608441    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"
Oct 30 18:52:23.995880 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:23.995852    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"]
Oct 30 18:52:23.999440 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:52:23.999411    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1d3787_e640_4027_b267_88a3c8a19873.slice/crio-ecf37a622416e5e45940bc30db8d625f44f2c35721b7f21eeb2e5e2072d2d75a WatchSource:0}: Error finding container ecf37a622416e5e45940bc30db8d625f44f2c35721b7f21eeb2e5e2072d2d75a: Status 404 returned error can't find the container with id ecf37a622416e5e45940bc30db8d625f44f2c35721b7f21eeb2e5e2072d2d75a
Oct 30 18:52:24.203403 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:24.203309    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs" event={"ID":"7c1d3787-e640-4027-b267-88a3c8a19873","Type":"ContainerStarted","Data":"1cdc9d72dfa04dd991fb99eb86ef431cacb95d8567f9531c8bc8fe5a4ebfc73b"}
Oct 30 18:52:24.203403 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:24.203380    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs" event={"ID":"7c1d3787-e640-4027-b267-88a3c8a19873","Type":"ContainerStarted","Data":"ecf37a622416e5e45940bc30db8d625f44f2c35721b7f21eeb2e5e2072d2d75a"}
Oct 30 18:52:24.203691 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:24.203409    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"
Oct 30 18:52:24.318876 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:24.318810    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs" podStartSLOduration=1.318789642 podStartE2EDuration="1.318789642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:24.316256345 +0000 UTC m=+782.959423960" watchObservedRunningTime="2026-10-30 18:52:24.318789642 +0000 UTC m=+782.961957256"
Oct 30 18:52:26.112729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:26.112700    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"
Oct 30 18:52:26.275614 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:26.275585    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:52:27.223909 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:27.223863    6014 generic.go:356] "Generic (PLEG): container finished" podID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerID="4ccad808aac5beeb6f8c43f2aa9661459394e46bb09aa63970586304dd693ab7" exitCode=0
Oct 30 18:52:27.224502 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:27.224096    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" event={"ID":"a13d3d6b-d787-4b0e-a46a-4a132174f16f","Type":"ContainerDied","Data":"4ccad808aac5beeb6f8c43f2aa9661459394e46bb09aa63970586304dd693ab7"}
Oct 30 18:52:27.224814 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:27.224792    6014 scope.go:122] "RemoveContainer" containerID="4ccad808aac5beeb6f8c43f2aa9661459394e46bb09aa63970586304dd693ab7"
Oct 30 18:52:27.226955 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:27.226908    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:27.227088 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:27.227062    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:27.227216 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:27.227195    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:28.075165 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.071455    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"]
Oct 30 18:52:28.090181 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.089905    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"
Oct 30 18:52:28.101164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.101089    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg\" (UID: \"7ee7e187-7e2a-4105-a4d0-1fe11a63464f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"
Oct 30 18:52:28.101356 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.101260    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79g8\" (UniqueName: \"kubernetes.io/projected/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-kube-api-access-c79g8\") pod \"payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg\" (UID: \"7ee7e187-7e2a-4105-a4d0-1fe11a63464f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"
Oct 30 18:52:28.125737 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.125705    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"
Oct 30 18:52:28.133295 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.133261    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"]
Oct 30 18:52:28.143184 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.143134    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f5763a-dockercfg-nhpkj\""
Oct 30 18:52:28.143388 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.143326    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f5763a\""
Oct 30 18:52:28.211166 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.210998    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg\" (UID: \"7ee7e187-7e2a-4105-a4d0-1fe11a63464f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"
Oct 30 18:52:28.211166 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.211126    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79g8\" (UniqueName: \"kubernetes.io/projected/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-kube-api-access-c79g8\") pod \"payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg\" (UID: \"7ee7e187-7e2a-4105-a4d0-1fe11a63464f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"
Oct 30 18:52:28.237669 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.237631    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg\" (UID: \"7ee7e187-7e2a-4105-a4d0-1fe11a63464f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"
Oct 30 18:52:28.239362 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.239333    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"]
Oct 30 18:52:28.243201 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.243173    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"
Oct 30 18:52:28.251884 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.251852    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" event={"ID":"a13d3d6b-d787-4b0e-a46a-4a132174f16f","Type":"ContainerStarted","Data":"a2b42546592e0b62e2796816651022c71baec4de51e3a1548d433fd47cf9c28e"}
Oct 30 18:52:28.306883 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.294548    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"]
Oct 30 18:52:28.308792 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.307846    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79g8\" (UniqueName: \"kubernetes.io/projected/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-kube-api-access-c79g8\") pod \"payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg\" (UID: \"7ee7e187-7e2a-4105-a4d0-1fe11a63464f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"
Oct 30 18:52:28.410412 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.410324    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"
Oct 30 18:52:28.412603 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.412580    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrwxg\" (UniqueName: \"kubernetes.io/projected/fe3f0683-26aa-408e-98d2-97d34bba0d14-kube-api-access-mrwxg\") pod \"payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk\" (UID: \"fe3f0683-26aa-408e-98d2-97d34bba0d14\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"
Oct 30 18:52:28.412929 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.412880    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe3f0683-26aa-408e-98d2-97d34bba0d14-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk\" (UID: \"fe3f0683-26aa-408e-98d2-97d34bba0d14\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"
Oct 30 18:52:28.514517 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.514486    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe3f0683-26aa-408e-98d2-97d34bba0d14-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk\" (UID: \"fe3f0683-26aa-408e-98d2-97d34bba0d14\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"
Oct 30 18:52:28.516107 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.514670    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrwxg\" (UniqueName: \"kubernetes.io/projected/fe3f0683-26aa-408e-98d2-97d34bba0d14-kube-api-access-mrwxg\") pod \"payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk\" (UID: \"fe3f0683-26aa-408e-98d2-97d34bba0d14\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"
Oct 30 18:52:28.516107 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.515617    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe3f0683-26aa-408e-98d2-97d34bba0d14-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk\" (UID: \"fe3f0683-26aa-408e-98d2-97d34bba0d14\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"
Oct 30 18:52:28.608718 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.608682    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrwxg\" (UniqueName: \"kubernetes.io/projected/fe3f0683-26aa-408e-98d2-97d34bba0d14-kube-api-access-mrwxg\") pod \"payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk\" (UID: \"fe3f0683-26aa-408e-98d2-97d34bba0d14\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"
Oct 30 18:52:28.622179 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.619324    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"
Oct 30 18:52:28.839694 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:28.839664    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"]
Oct 30 18:52:28.842224 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:52:28.842199    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee7e187_7e2a_4105_a4d0_1fe11a63464f.slice/crio-b3b3d229e1989c5f1be0edb36439f5e6536820f64dfe946bedfaef59da7fb726 WatchSource:0}: Error finding container b3b3d229e1989c5f1be0edb36439f5e6536820f64dfe946bedfaef59da7fb726: Status 404 returned error can't find the container with id b3b3d229e1989c5f1be0edb36439f5e6536820f64dfe946bedfaef59da7fb726
Oct 30 18:52:29.039209 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:29.039179    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"]
Oct 30 18:52:29.042354 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:29.042326    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:52:29.042577 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:52:29.042556    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3f0683_26aa_408e_98d2_97d34bba0d14.slice/crio-6027679de4c93140058bc6d85a9113fde4c8bc6f5b872a518f2d08ca27325d6e WatchSource:0}: Error finding container 6027679de4c93140058bc6d85a9113fde4c8bc6f5b872a518f2d08ca27325d6e: Status 404 returned error can't find the container with id 6027679de4c93140058bc6d85a9113fde4c8bc6f5b872a518f2d08ca27325d6e
Oct 30 18:52:29.043285 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:29.043264    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:52:29.043369 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:29.043314    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 18:52:29.300249 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:29.299797    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk" event={"ID":"fe3f0683-26aa-408e-98d2-97d34bba0d14","Type":"ContainerStarted","Data":"478079fb93390826deb769516568bddaef7996d9733160bb57e715b62d18d80f"}
Oct 30 18:52:29.300249 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:29.299870    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk" event={"ID":"fe3f0683-26aa-408e-98d2-97d34bba0d14","Type":"ContainerStarted","Data":"6027679de4c93140058bc6d85a9113fde4c8bc6f5b872a518f2d08ca27325d6e"}
Oct 30 18:52:29.300249 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:29.299915    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"
Oct 30 18:52:29.303485 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:29.303452    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg" event={"ID":"7ee7e187-7e2a-4105-a4d0-1fe11a63464f","Type":"ContainerStarted","Data":"24414dbda1894e4effc481934606d8b1c710c724f3612a40bc2bf2011a353c88"}
Oct 30 18:52:29.303635 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:29.303516    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg" event={"ID":"7ee7e187-7e2a-4105-a4d0-1fe11a63464f","Type":"ContainerStarted","Data":"b3b3d229e1989c5f1be0edb36439f5e6536820f64dfe946bedfaef59da7fb726"}
Oct 30 18:52:29.303716 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:29.303691    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"
Oct 30 18:52:29.434227 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:29.434177    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk" podStartSLOduration=1.434162321 podStartE2EDuration="1.434162321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:29.432400208 +0000 UTC m=+788.075567824" watchObservedRunningTime="2026-10-30 18:52:29.434162321 +0000 UTC m=+788.077329935"
Oct 30 18:52:29.579171 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:29.578277    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg" podStartSLOduration=1.578252507 podStartE2EDuration="1.578252507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:29.564326589 +0000 UTC m=+788.207494205" watchObservedRunningTime="2026-10-30 18:52:29.578252507 +0000 UTC m=+788.221420122"
Oct 30 18:52:30.042949 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:30.042873    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:52:30.043187 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:30.043128    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 18:52:30.812170 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:30.805023    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"]
Oct 30 18:52:30.821041 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:30.821013    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"
Oct 30 18:52:30.858236 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:30.847421    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"]
Oct 30 18:52:30.861174 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:30.859655    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7b5s\" (UniqueName: \"kubernetes.io/projected/33c08345-1df9-4df2-883d-8715226336b3-kube-api-access-j7b5s\") pod \"payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps\" (UID: \"33c08345-1df9-4df2-883d-8715226336b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"
Oct 30 18:52:30.861174 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:30.859803    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c08345-1df9-4df2-883d-8715226336b3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps\" (UID: \"33c08345-1df9-4df2-883d-8715226336b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"
Oct 30 18:52:30.890644 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:30.890607    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-bc98e1\""
Oct 30 18:52:30.910446 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:30.895174    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-bc98e1-dockercfg-9wbvg\""
Oct 30 18:52:30.983166 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:30.982164    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7b5s\" (UniqueName: \"kubernetes.io/projected/33c08345-1df9-4df2-883d-8715226336b3-kube-api-access-j7b5s\") pod \"payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps\" (UID: \"33c08345-1df9-4df2-883d-8715226336b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"
Oct 30 18:52:30.983166 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:30.982288    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c08345-1df9-4df2-883d-8715226336b3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps\" (UID: \"33c08345-1df9-4df2-883d-8715226336b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"
Oct 30 18:52:30.993167 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:30.991413    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c08345-1df9-4df2-883d-8715226336b3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps\" (UID: \"33c08345-1df9-4df2-883d-8715226336b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"
Oct 30 18:52:31.043224 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:31.043188    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:52:31.043382 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:31.043246    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 18:52:31.044110 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:31.044088    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7b5s\" (UniqueName: \"kubernetes.io/projected/33c08345-1df9-4df2-883d-8715226336b3-kube-api-access-j7b5s\") pod \"payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps\" (UID: \"33c08345-1df9-4df2-883d-8715226336b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"
Oct 30 18:52:31.189165 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:31.189056    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"
Oct 30 18:52:31.602095 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:31.599919    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"]
Oct 30 18:52:31.606305 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:52:31.606271    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c08345_1df9_4df2_883d_8715226336b3.slice/crio-7c9132041741b9daa540c6207450c5cdf0922a3c5bbaf235e83845bb226f24e6 WatchSource:0}: Error finding container 7c9132041741b9daa540c6207450c5cdf0922a3c5bbaf235e83845bb226f24e6: Status 404 returned error can't find the container with id 7c9132041741b9daa540c6207450c5cdf0922a3c5bbaf235e83845bb226f24e6
Oct 30 18:52:32.011524 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:32.011435    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:52:32.043277 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:32.043246    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:52:32.043435 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:32.043293    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 18:52:32.332888 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:32.332854    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps" event={"ID":"33c08345-1df9-4df2-883d-8715226336b3","Type":"ContainerStarted","Data":"02f6a1d4ed248c951e99d4a69e4a903f5a8caf786ee0f8ed0c3a8041a47f2e3d"}
Oct 30 18:52:32.333117 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:32.332910    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps" event={"ID":"33c08345-1df9-4df2-883d-8715226336b3","Type":"ContainerStarted","Data":"7c9132041741b9daa540c6207450c5cdf0922a3c5bbaf235e83845bb226f24e6"}
Oct 30 18:52:32.333117 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:32.332944    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"
Oct 30 18:52:32.455171 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:32.449051    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps" podStartSLOduration=2.449032003 podStartE2EDuration="2.449032003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:32.446864297 +0000 UTC m=+791.090031912" watchObservedRunningTime="2026-10-30 18:52:32.449032003 +0000 UTC m=+791.092199619"
Oct 30 18:52:33.042757 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:33.042644    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:52:33.042757 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:33.042730    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 18:52:33.199257 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:33.199190    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"
Oct 30 18:52:34.047193 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:34.047164    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:52:34.048322 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:34.048298    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 18:52:34.057289 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:34.057261    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:52:34.057478 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:34.057323    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:35.213773 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:35.213741    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"
Oct 30 18:52:36.360740 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:36.360642    6014 generic.go:356] "Generic (PLEG): container finished" podID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerID="01c0757f95ab0889c11dbb6afdaef8757d59c628a0b07e3c719bad68138d0dba" exitCode=0
Oct 30 18:52:36.360740 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:36.360682    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" event={"ID":"cab5db1f-c822-4fe8-a692-6e8aaa8306bb","Type":"ContainerDied","Data":"01c0757f95ab0889c11dbb6afdaef8757d59c628a0b07e3c719bad68138d0dba"}
Oct 30 18:52:36.361383 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:36.361170    6014 scope.go:122] "RemoveContainer" containerID="01c0757f95ab0889c11dbb6afdaef8757d59c628a0b07e3c719bad68138d0dba"
Oct 30 18:52:36.362496 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:36.362463    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:36.362582 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:36.362532    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:36.362582 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:36.362564    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:37.377307 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:37.377273    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" event={"ID":"cab5db1f-c822-4fe8-a692-6e8aaa8306bb","Type":"ContainerStarted","Data":"84452e74be4caaf71b7a03af6561f7ba06e30d6fa7fbf9c1b88d13394a33d413"}
Oct 30 18:52:38.002409 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:38.002371    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:38.002661 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:38.002636    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:52:38.002745 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:38.002710    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 18:52:39.002319 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:39.002288    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:52:39.002723 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:39.002338    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 18:52:39.457167 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:39.455534    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:52:40.002184 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:40.002113    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:52:40.002357 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:40.002212    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 18:52:40.312850 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:40.312765    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"
Oct 30 18:52:41.002459 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:41.002417    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:52:41.002875 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:41.002477    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 18:52:41.321082 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:41.321058    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"
Oct 30 18:52:42.003031 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:42.002997    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:52:42.003480 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:42.003055    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 18:52:43.006220 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:43.006191    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:43.006725 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:43.006567    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:43.007272 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:43.007253    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 18:52:43.343814 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:43.343788    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"
Oct 30 18:52:49.733355 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:49.733318    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:52:53.281887 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:52:53.281852    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:52:55.136214 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.136182    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"]
Oct 30 18:52:55.141323 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.141303    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.215128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.215088    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-metrics-certs\") pod \"maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.215312 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.215279    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.215379 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.215338    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btdnj\" (UniqueName: \"kubernetes.io/projected/1456cdbd-5d69-4731-9da9-6b2e803be1f9-kube-api-access-btdnj\") pod \"maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.215443 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.215412    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1456cdbd-5d69-4731-9da9-6b2e803be1f9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.244514 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.244481    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:52:55.244746 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.244726    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9wflp\""
Oct 30 18:52:55.244814 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.244767    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-bc98e1-serving-cert\""
Oct 30 18:52:55.244814 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.244809    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 18:52:55.245163 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.245113    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:52:55.247112 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.247050    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"]
Oct 30 18:52:55.316563 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.316521    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-metrics-certs\") pod \"maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.316803 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.316618    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.316803 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.316665    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdnj\" (UniqueName: \"kubernetes.io/projected/1456cdbd-5d69-4731-9da9-6b2e803be1f9-kube-api-access-btdnj\") pod \"maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.316803 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.316701    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1456cdbd-5d69-4731-9da9-6b2e803be1f9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.317348 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.317325    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1456cdbd-5d69-4731-9da9-6b2e803be1f9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.319043 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.319025    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-metrics-certs\") pod \"maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.319184 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.319132    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.375692 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.375657    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdnj\" (UniqueName: \"kubernetes.io/projected/1456cdbd-5d69-4731-9da9-6b2e803be1f9-kube-api-access-btdnj\") pod \"maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.495205 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.495088    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:52:55.909474 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:55.909448    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"]
Oct 30 18:52:55.912335 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:52:55.912301    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1456cdbd_5d69_4731_9da9_6b2e803be1f9.slice/crio-ec08c99c93f4802bf701d850d61353a6496c2799ae19fa7616888ec59138caac WatchSource:0}: Error finding container ec08c99c93f4802bf701d850d61353a6496c2799ae19fa7616888ec59138caac: Status 404 returned error can't find the container with id ec08c99c93f4802bf701d850d61353a6496c2799ae19fa7616888ec59138caac
Oct 30 18:52:56.520122 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:56.520081    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv" event={"ID":"1456cdbd-5d69-4731-9da9-6b2e803be1f9","Type":"ContainerStarted","Data":"ec08c99c93f4802bf701d850d61353a6496c2799ae19fa7616888ec59138caac"}
Oct 30 18:52:58.247679 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:58.247645    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:52:59.544208 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:52:59.544172    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv" event={"ID":"1456cdbd-5d69-4731-9da9-6b2e803be1f9","Type":"ContainerStarted","Data":"66edfd02157e2136f15f3bb7ecb6d8176e86c77cb421afc5fa6ed1df37adcf3e"}
Oct 30 18:53:00.563221 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:00.562801    6014 generic.go:356] "Generic (PLEG): container finished" podID="945375bc-250a-4b52-8cfc-e405096baf26" containerID="a42f5e791085fd50c3401e21930ab7422508d5b44609144db981f97a1363bd7e" exitCode=0
Oct 30 18:53:00.565414 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:00.564342    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" event={"ID":"945375bc-250a-4b52-8cfc-e405096baf26","Type":"ContainerDied","Data":"a42f5e791085fd50c3401e21930ab7422508d5b44609144db981f97a1363bd7e"}
Oct 30 18:53:00.565414 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:00.564403    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:53:00.565414 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:00.564770    6014 scope.go:122] "RemoveContainer" containerID="a42f5e791085fd50c3401e21930ab7422508d5b44609144db981f97a1363bd7e"
Oct 30 18:53:00.568275 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:00.567757    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:00.568275 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:00.567846    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:00.568275 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:00.567894    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:00.783264 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:00.783195    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv" podStartSLOduration=2.609523489 podStartE2EDuration="5.783131504s" totalImagesPullingTime="3.173608015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:59.671365729 +0000 UTC m=+818.314533343" watchObservedRunningTime="2026-10-30 18:53:00.783131504 +0000 UTC m=+819.426299119"
Oct 30 18:53:01.600165 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:01.599176    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" event={"ID":"945375bc-250a-4b52-8cfc-e405096baf26","Type":"ContainerStarted","Data":"139b673c801b033b8b72f17bbfa823fcd6b1c91daf108eacadc638bc119990c4"}
Oct 30 18:53:02.339469 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:02.339426    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:53:02.340385 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:02.340362    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:53:02.340449 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:02.340429    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" podUID="945375bc-250a-4b52-8cfc-e405096baf26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 18:53:03.016037 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:03.016000    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:53:03.339237 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:03.339197    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:53:03.339430 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:03.339264    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" podUID="945375bc-250a-4b52-8cfc-e405096baf26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 18:53:04.339710 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:04.339657    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:53:04.340131 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:04.339728    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" podUID="945375bc-250a-4b52-8cfc-e405096baf26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 18:53:05.339927 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:05.339890    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:53:05.340351 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:05.339952    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" podUID="945375bc-250a-4b52-8cfc-e405096baf26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 18:53:06.339164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:06.339106    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:53:06.339355 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:06.339172    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" podUID="945375bc-250a-4b52-8cfc-e405096baf26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 18:53:06.606249 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:06.606126    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 18:53:07.344330 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:07.344286    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:53:07.345463 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:07.345438    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 18:53:08.661574 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:08.661536    6014 generic.go:356] "Generic (PLEG): container finished" podID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerID="e5266cc0ab19505eff5c3e1dc325c6fc1b59970fac3e54aa6c1aa2fb765e62df" exitCode=0
Oct 30 18:53:08.662120 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:08.661612    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4" event={"ID":"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84","Type":"ContainerDied","Data":"e5266cc0ab19505eff5c3e1dc325c6fc1b59970fac3e54aa6c1aa2fb765e62df"}
Oct 30 18:53:08.662234 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:08.662123    6014 scope.go:122] "RemoveContainer" containerID="e5266cc0ab19505eff5c3e1dc325c6fc1b59970fac3e54aa6c1aa2fb765e62df"
Oct 30 18:53:08.663495 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:08.663464    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:08.663659 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:08.663548    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:08.663659 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:08.663597    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:09.680565 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:09.680521    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4" event={"ID":"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84","Type":"ContainerStarted","Data":"7e1134d28476871e6715914d521086e1a3a55ac3f76e0eab912ece02d4a0ecd3"}
Oct 30 18:53:10.677322 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:10.677279    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:53:10.677551 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:10.677531    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:53:10.677626 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:10.677590    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4" podUID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 18:53:11.677707 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:11.677660    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:53:11.678252 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:11.677714    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4" podUID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 18:53:12.677702 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:12.677669    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:53:12.677904 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:12.677736    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4" podUID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 18:53:13.677525 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:13.677488    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:53:13.677713 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:13.677541    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4" podUID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 18:53:14.031996 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:14.031914    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:53:14.677705 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:14.677657    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:53:14.677889 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:14.677732    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4" podUID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 18:53:15.681523 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:15.681491    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:53:15.681926 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:15.681851    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:53:15.682597 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:15.682580    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 18:53:23.259975 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:53:23.259939    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:53:33.234621 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.234587    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"]
Oct 30 18:53:33.240185 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.240133    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.341835 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.341795    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-dshm\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.342041 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.341854    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-home\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.342041 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.341950    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-tmp-dir\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.342041 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.341988    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/664bad09-b36f-46cd-8970-034c6094bd65-tls-certs\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.342229 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.342182    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-model-cache\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.342229 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.342208    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gsq\" (UniqueName: \"kubernetes.io/projected/664bad09-b36f-46cd-8970-034c6094bd65-kube-api-access-v5gsq\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.343685 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.343646    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-975cf5\"/\"default-dockercfg-5h568\""
Oct 30 18:53:33.343685 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.343656    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-975cf5\"/\"simulator-w1-975cf5-kserve-self-signed-certs\""
Oct 30 18:53:33.343875 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.343705    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-975cf5\"/\"kube-root-ca.crt\""
Oct 30 18:53:33.343875 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.343748    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-975cf5\"/\"openshift-service-ca.crt\""
Oct 30 18:53:33.345759 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.345739    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"]
Oct 30 18:53:33.443051 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.443016    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-dshm\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.443275 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.443068    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-home\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.443275 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.443117    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-tmp-dir\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.443275 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.443194    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/664bad09-b36f-46cd-8970-034c6094bd65-tls-certs\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.443456 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.443347    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-model-cache\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.443456 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.443378    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gsq\" (UniqueName: \"kubernetes.io/projected/664bad09-b36f-46cd-8970-034c6094bd65-kube-api-access-v5gsq\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.443586 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.443565    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-home\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.443652 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.443630    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-tmp-dir\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.443745 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.443726    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-model-cache\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.445867 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.445818    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-dshm\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.446418 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.446395    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/664bad09-b36f-46cd-8970-034c6094bd65-tls-certs\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.506604 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.506526    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gsq\" (UniqueName: \"kubernetes.io/projected/664bad09-b36f-46cd-8970-034c6094bd65-kube-api-access-v5gsq\") pod \"simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") " pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.552642 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.552597    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:33.934876 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:33.934851    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"]
Oct 30 18:53:33.937323 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:53:33.937290    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod664bad09_b36f_46cd_8970_034c6094bd65.slice/crio-ec82ec85e401b7455d447afa402e9f824905630ace7bbb42f7d46c69409d4417 WatchSource:0}: Error finding container ec82ec85e401b7455d447afa402e9f824905630ace7bbb42f7d46c69409d4417: Status 404 returned error can't find the container with id ec82ec85e401b7455d447afa402e9f824905630ace7bbb42f7d46c69409d4417
Oct 30 18:53:34.827100 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:34.827059    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" event={"ID":"664bad09-b36f-46cd-8970-034c6094bd65","Type":"ContainerStarted","Data":"34eeca231dccd2dda7023d5d343c84802dcbf9ade4ddf306e45e33eb2270abed"}
Oct 30 18:53:34.827100 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:34.827105    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" event={"ID":"664bad09-b36f-46cd-8970-034c6094bd65","Type":"ContainerStarted","Data":"ec82ec85e401b7455d447afa402e9f824905630ace7bbb42f7d46c69409d4417"}
Oct 30 18:53:35.553271 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:35.553233    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:39.070778 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:39.070721    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podStartSLOduration=6.070701397 podStartE2EDuration="6.070701397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:34.952782788 +0000 UTC m=+853.595950402" watchObservedRunningTime="2026-10-30 18:53:39.070701397 +0000 UTC m=+857.713869011"
Oct 30 18:53:39.072147 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:39.072118    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:53:43.553353 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:43.553323    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:43.572420 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:43.572393    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:43.891125 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:43.891094    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 18:53:53.258509 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:53:53.258466    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:53:57.358019 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.357978    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"]
Oct 30 18:53:57.362409 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.362384    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.418800 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.418772    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-611053\"/\"openshift-service-ca.crt\""
Oct 30 18:53:57.418970 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.418819    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-611053\"/\"default-dockercfg-h72q7\""
Oct 30 18:53:57.468384 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.468347    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-611053\"/\"kube-root-ca.crt\""
Oct 30 18:53:57.468575 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.468383    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-611053\"/\"simulator-w5-611053-kserve-self-signed-certs\""
Oct 30 18:53:57.475831 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.475803    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"]
Oct 30 18:53:57.523454 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.523422    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-home\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.523644 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.523518    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-model-cache\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.523644 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.523599    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-dshm\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.523644 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.523623    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-tmp-dir\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.523768 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.523642    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqtq7\" (UniqueName: \"kubernetes.io/projected/63526e12-eb15-4369-a74c-8b77f983da8d-kube-api-access-tqtq7\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.523768 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.523665    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63526e12-eb15-4369-a74c-8b77f983da8d-tls-certs\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.625029 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.624914    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-model-cache\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.625029 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.625008    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-dshm\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.625451 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.625051    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-tmp-dir\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.625451 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.625076    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqtq7\" (UniqueName: \"kubernetes.io/projected/63526e12-eb15-4369-a74c-8b77f983da8d-kube-api-access-tqtq7\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.625451 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.625106    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63526e12-eb15-4369-a74c-8b77f983da8d-tls-certs\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.625451 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.625253    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-home\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.625770 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.625547    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-tmp-dir\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.625770 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.625671    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-home\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.626250 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.626220    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-model-cache\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.627943 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.627923    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-dshm\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.628072 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.628055    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63526e12-eb15-4369-a74c-8b77f983da8d-tls-certs\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.684568 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.684535    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqtq7\" (UniqueName: \"kubernetes.io/projected/63526e12-eb15-4369-a74c-8b77f983da8d-kube-api-access-tqtq7\") pod \"simulator-w5-611053-kserve-6b94f9d85b-82447\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") " pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:57.976105 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:57.976020    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:58.353020 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:58.352995    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"]
Oct 30 18:53:58.355355 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:53:58.355324    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63526e12_eb15_4369_a74c_8b77f983da8d.slice/crio-9eb891647ab8b9d5200d9a842773b816254e99c328645483e2b5be0c135ac4d5 WatchSource:0}: Error finding container 9eb891647ab8b9d5200d9a842773b816254e99c328645483e2b5be0c135ac4d5: Status 404 returned error can't find the container with id 9eb891647ab8b9d5200d9a842773b816254e99c328645483e2b5be0c135ac4d5
Oct 30 18:53:58.969569 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:58.969535    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" event={"ID":"63526e12-eb15-4369-a74c-8b77f983da8d","Type":"ContainerStarted","Data":"43f9da463fc74873ce353be3150ac1f7e17d41967e007ca8dd4a2c84fa2d14ea"}
Oct 30 18:53:58.969569 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:58.969577    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" event={"ID":"63526e12-eb15-4369-a74c-8b77f983da8d","Type":"ContainerStarted","Data":"9eb891647ab8b9d5200d9a842773b816254e99c328645483e2b5be0c135ac4d5"}
Oct 30 18:53:58.976898 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:58.976869    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:53:59.090958 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:53:59.090907    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podStartSLOduration=2.090891751 podStartE2EDuration="2.090891751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:59.088365971 +0000 UTC m=+877.731533585" watchObservedRunningTime="2026-10-30 18:53:59.090891751 +0000 UTC m=+877.734059364"
Oct 30 18:54:03.692405 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:03.692369    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:54:07.976888 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:07.976840    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:54:07.989513 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:07.989490    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:54:08.032980 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:08.032952    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 18:54:23.258453 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:54:23.258408    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:54:26.883917 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:26.883886    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"]
Oct 30 18:54:26.888394 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:26.888371    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:26.963217 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:26.963177    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-home\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:26.963393 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:26.963243    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tls-certs\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:26.963393 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:26.963324    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tmp-dir\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:26.963393 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:26.963366    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-dshm\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:26.963568 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:26.963543    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-model-cache\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:26.963683 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:26.963665    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xtk7\" (UniqueName: \"kubernetes.io/projected/9c84976c-cb92-4436-99a4-d93ef5cd07d4-kube-api-access-6xtk7\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:26.990503 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:26.990451    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bc98e1\"/\"default-dockercfg-hc99l\""
Oct 30 18:54:26.990503 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:26.990469    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bc98e1\"/\"openshift-service-ca.crt\""
Oct 30 18:54:26.990503 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:26.990459    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bc98e1\"/\"simulator-w3-bc98e1-kserve-self-signed-certs\""
Oct 30 18:54:26.990503 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:26.990506    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bc98e1\"/\"kube-root-ca.crt\""
Oct 30 18:54:26.992724 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:26.992707    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"]
Oct 30 18:54:27.064696 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.064663    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xtk7\" (UniqueName: \"kubernetes.io/projected/9c84976c-cb92-4436-99a4-d93ef5cd07d4-kube-api-access-6xtk7\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:27.064880 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.064734    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-home\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:27.064880 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.064766    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tls-certs\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:27.064880 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.064785    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tmp-dir\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:27.065019 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.064900    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-dshm\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:27.065069 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.065012    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-model-cache\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:27.065258 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.065237    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-home\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:27.065355 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.065287    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tmp-dir\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:27.065400 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.065392    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-model-cache\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:27.067343 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.067285    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-dshm\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:27.067546 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.067531    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tls-certs\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:27.123833 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.123797    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xtk7\" (UniqueName: \"kubernetes.io/projected/9c84976c-cb92-4436-99a4-d93ef5cd07d4-kube-api-access-6xtk7\") pod \"simulator-w3-bc98e1-kserve-76bc58878-bz45z\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") " pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:27.200294 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.200192    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:27.578567 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:27.578536    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"]
Oct 30 18:54:27.582392 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:54:27.582361    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c84976c_cb92_4436_99a4_d93ef5cd07d4.slice/crio-caa1e8037569e5211a3b4f3e74c63434c5331b8dfd921fc6f601097df7cfa04a WatchSource:0}: Error finding container caa1e8037569e5211a3b4f3e74c63434c5331b8dfd921fc6f601097df7cfa04a: Status 404 returned error can't find the container with id caa1e8037569e5211a3b4f3e74c63434c5331b8dfd921fc6f601097df7cfa04a
Oct 30 18:54:28.143501 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:28.143454    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" event={"ID":"9c84976c-cb92-4436-99a4-d93ef5cd07d4","Type":"ContainerStarted","Data":"662222cc5bec55f92cfae55c3cc02324f61296564a817d3e106817b1eb7e34dd"}
Oct 30 18:54:28.143501 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:28.143491    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" event={"ID":"9c84976c-cb92-4436-99a4-d93ef5cd07d4","Type":"ContainerStarted","Data":"caa1e8037569e5211a3b4f3e74c63434c5331b8dfd921fc6f601097df7cfa04a"}
Oct 30 18:54:28.200658 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:28.200628    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:28.262538 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:28.262484    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podStartSLOduration=2.262469778 podStartE2EDuration="2.262469778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:28.25995087 +0000 UTC m=+906.903118484" watchObservedRunningTime="2026-10-30 18:54:28.262469778 +0000 UTC m=+906.905637391"
Oct 30 18:54:31.556390 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:31.556306    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:54:37.200535 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:37.200496    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:37.213052 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:37.213019    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:38.209703 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:38.209673    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 18:54:46.890618 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:46.890588    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"]
Oct 30 18:54:46.894868 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:46.894845    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:46.946695 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:46.946670    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-66bdc0-openshift-default-dockercfg-d69f7\""
Oct 30 18:54:46.998262 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:46.998223    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"]
Oct 30 18:54:47.002710 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.002681    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.002839 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.002727    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.002839 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.002750    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.002921 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.002854    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.002921 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.002882    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.002986 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.002962    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.003020 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.002985    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.003020 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.003004    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k29s\" (UniqueName: \"kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-kube-api-access-6k29s\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.003109 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.003037    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.104769 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.104735    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.104769 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.104774    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.105022 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.104794    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.105022 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.104827    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.105022 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.104844    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.105022 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.104888    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.105022 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.104905    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.105022 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.104920    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k29s\" (UniqueName: \"kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-kube-api-access-6k29s\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.105022 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.104956    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.105459 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.105285    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.105459 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.105345    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.105740 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.105659    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.105740 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.105732    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.105740 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.105768    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.107388 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.107369    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.107668 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.107650    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.167705 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.167628    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.168348 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.168320    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k29s\" (UniqueName: \"kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-kube-api-access-6k29s\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.208952 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.208917    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:47.649081 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.649053    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"]
Oct 30 18:54:47.650603 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:54:47.650572    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b35af62_6e06_4f5e_8c63_4b8a479f6845.slice/crio-058efba49120a3f17729e7cffedbbab1c104ebcaebca61d951a6b2be10d74e81 WatchSource:0}: Error finding container 058efba49120a3f17729e7cffedbbab1c104ebcaebca61d951a6b2be10d74e81: Status 404 returned error can't find the container with id 058efba49120a3f17729e7cffedbbab1c104ebcaebca61d951a6b2be10d74e81
Oct 30 18:54:47.653189 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.653111    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:54:47.653289 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.653220    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:54:47.653289 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:47.653268    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:54:48.276939 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:48.276903    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng" event={"ID":"2b35af62-6e06-4f5e-8c63-4b8a479f6845","Type":"ContainerStarted","Data":"4ec00da1183b450ccb3e4a43f23b5aa198a22d5988a0daade0aaeffacf39c377"}
Oct 30 18:54:48.277472 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:48.276972    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng" event={"ID":"2b35af62-6e06-4f5e-8c63-4b8a479f6845","Type":"ContainerStarted","Data":"058efba49120a3f17729e7cffedbbab1c104ebcaebca61d951a6b2be10d74e81"}
Oct 30 18:54:48.401337 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:48.401289    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng" podStartSLOduration=2.401272518 podStartE2EDuration="2.401272518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:48.39933462 +0000 UTC m=+927.042502234" watchObservedRunningTime="2026-10-30 18:54:48.401272518 +0000 UTC m=+927.044440154"
Oct 30 18:54:49.210343 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:49.210298    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:49.211995 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:49.211968    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:54:49.212160 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:49.212022    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng" podUID="2b35af62-6e06-4f5e-8c63-4b8a479f6845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 18:54:50.209896 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:50.209856    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:54:50.210383 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:50.209920    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng" podUID="2b35af62-6e06-4f5e-8c63-4b8a479f6845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 18:54:51.210662 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:51.210625    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:54:51.211081 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:51.210688    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng" podUID="2b35af62-6e06-4f5e-8c63-4b8a479f6845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 18:54:52.210805 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:52.210765    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:54:52.211275 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:52.210836    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng" podUID="2b35af62-6e06-4f5e-8c63-4b8a479f6845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 18:54:53.210034 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:53.210002    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:54:53.210249 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:53.210056    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng" podUID="2b35af62-6e06-4f5e-8c63-4b8a479f6845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 18:54:53.282076 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:54:53.282039    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:54:54.209521 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:54.209491    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:54:54.209718 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:54.209549    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng" podUID="2b35af62-6e06-4f5e-8c63-4b8a479f6845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 18:54:55.062422 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:55.062384    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:54:55.215152 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:55.215113    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:55.215623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:55.215604    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:55.216377 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:55.216362    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 18:54:56.906772 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:56.906738    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"]
Oct 30 18:54:56.910972 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:56.910947    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"
Oct 30 18:54:56.963095 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:56.963063    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:54:57.012493 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.012458    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:54:57.014656 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.014635    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"]
Oct 30 18:54:57.047442 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.047411    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r9wf\" (UniqueName: \"kubernetes.io/projected/774d9002-4d61-4dba-a869-3ef9353d170a-kube-api-access-8r9wf\") pod \"maas-api-revoke-sub-06a16b51-txbrl\" (UID: \"774d9002-4d61-4dba-a869-3ef9353d170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"
Oct 30 18:54:57.047617 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.047463    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a16b51-txbrl\" (UID: \"774d9002-4d61-4dba-a869-3ef9353d170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"
Oct 30 18:54:57.047617 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.047587    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a16b51-txbrl\" (UID: \"774d9002-4d61-4dba-a869-3ef9353d170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"
Oct 30 18:54:57.148789 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.148755    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r9wf\" (UniqueName: \"kubernetes.io/projected/774d9002-4d61-4dba-a869-3ef9353d170a-kube-api-access-8r9wf\") pod \"maas-api-revoke-sub-06a16b51-txbrl\" (UID: \"774d9002-4d61-4dba-a869-3ef9353d170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"
Oct 30 18:54:57.148980 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.148810    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a16b51-txbrl\" (UID: \"774d9002-4d61-4dba-a869-3ef9353d170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"
Oct 30 18:54:57.148980 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.148856    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a16b51-txbrl\" (UID: \"774d9002-4d61-4dba-a869-3ef9353d170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"
Oct 30 18:54:57.149554 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.149531    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a16b51-txbrl\" (UID: \"774d9002-4d61-4dba-a869-3ef9353d170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"
Oct 30 18:54:57.151551 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.151530    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a16b51-txbrl\" (UID: \"774d9002-4d61-4dba-a869-3ef9353d170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"
Oct 30 18:54:57.206512 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.206443    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r9wf\" (UniqueName: \"kubernetes.io/projected/774d9002-4d61-4dba-a869-3ef9353d170a-kube-api-access-8r9wf\") pod \"maas-api-revoke-sub-06a16b51-txbrl\" (UID: \"774d9002-4d61-4dba-a869-3ef9353d170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"
Oct 30 18:54:57.224890 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.224864    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"
Oct 30 18:54:57.602514 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.602492    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"]
Oct 30 18:54:57.605059 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:54:57.605030    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774d9002_4d61_4dba_a869_3ef9353d170a.slice/crio-4872999b7c66315cb4247511edb90e0d05e1cb6dfbc0209d18e303ea372e8db2 WatchSource:0}: Error finding container 4872999b7c66315cb4247511edb90e0d05e1cb6dfbc0209d18e303ea372e8db2: Status 404 returned error can't find the container with id 4872999b7c66315cb4247511edb90e0d05e1cb6dfbc0209d18e303ea372e8db2
Oct 30 18:54:57.607103 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:57.607084    6014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:54:58.339619 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:54:58.339581    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl" event={"ID":"774d9002-4d61-4dba-a869-3ef9353d170a","Type":"ContainerStarted","Data":"4872999b7c66315cb4247511edb90e0d05e1cb6dfbc0209d18e303ea372e8db2"}
Oct 30 18:55:02.365210 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:02.365177    6014 generic.go:356] "Generic (PLEG): container finished" podID="774d9002-4d61-4dba-a869-3ef9353d170a" containerID="35702e58799122a7030f73e81085391d66e92db7ffcc113fc5b5f86c49fc0a1e" exitCode=0
Oct 30 18:55:02.365603 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:02.365255    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl" event={"ID":"774d9002-4d61-4dba-a869-3ef9353d170a","Type":"ContainerDied","Data":"35702e58799122a7030f73e81085391d66e92db7ffcc113fc5b5f86c49fc0a1e"}
Oct 30 18:55:03.522588 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:03.522558    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"
Oct 30 18:55:03.620950 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:03.620914    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774d9002-4d61-4dba-a869-3ef9353d170a-kube-api-access-8r9wf\" (UniqueName: \"kubernetes.io/projected/774d9002-4d61-4dba-a869-3ef9353d170a-kube-api-access-8r9wf\") pod \"774d9002-4d61-4dba-a869-3ef9353d170a\" (UID: \"774d9002-4d61-4dba-a869-3ef9353d170a\") "
Oct 30 18:55:03.621121 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:03.621020    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-openshift-ca\") pod \"774d9002-4d61-4dba-a869-3ef9353d170a\" (UID: \"774d9002-4d61-4dba-a869-3ef9353d170a\") "
Oct 30 18:55:03.621121 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:03.621113    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-xks-ca\") pod \"774d9002-4d61-4dba-a869-3ef9353d170a\" (UID: \"774d9002-4d61-4dba-a869-3ef9353d170a\") "
Oct 30 18:55:03.621403 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:03.621376    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-openshift-ca" pod "774d9002-4d61-4dba-a869-3ef9353d170a" (UID: "774d9002-4d61-4dba-a869-3ef9353d170a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:03.623449 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:03.623387    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-xks-ca" pod "774d9002-4d61-4dba-a869-3ef9353d170a" (UID: "774d9002-4d61-4dba-a869-3ef9353d170a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:03.623607 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:03.623586    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774d9002-4d61-4dba-a869-3ef9353d170a-kube-api-access-8r9wf" pod "774d9002-4d61-4dba-a869-3ef9353d170a" (UID: "774d9002-4d61-4dba-a869-3ef9353d170a"). InnerVolumeSpecName "kube-api-access-8r9wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:03.722912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:03.722875    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:03.722912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:03.722908    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774d9002-4d61-4dba-a869-3ef9353d170a-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:03.722912 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:03.722920    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r9wf\" (UniqueName: \"kubernetes.io/projected/774d9002-4d61-4dba-a869-3ef9353d170a-kube-api-access-8r9wf\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:04.387896 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:04.387864    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"
Oct 30 18:55:04.388085 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:04.387893    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl" event={"ID":"774d9002-4d61-4dba-a869-3ef9353d170a","Type":"ContainerDied","Data":"4872999b7c66315cb4247511edb90e0d05e1cb6dfbc0209d18e303ea372e8db2"}
Oct 30 18:55:04.388085 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:04.387962    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4872999b7c66315cb4247511edb90e0d05e1cb6dfbc0209d18e303ea372e8db2"
Oct 30 18:55:12.016828 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.016791    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"]
Oct 30 18:55:12.017382 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.017369    6014 cpu_manager.go:414] "containerMap: removing container" podUID="774d9002-4d61-4dba-a869-3ef9353d170a" containerName="revoke-keys"
Oct 30 18:55:12.017431 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.017383    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774d9002-4d61-4dba-a869-3ef9353d170a" containerName="revoke-keys"
Oct 30 18:55:12.017511 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.017502    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="774d9002-4d61-4dba-a869-3ef9353d170a" containerName="revoke-keys"
Oct 30 18:55:12.020729 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.020713    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"
Oct 30 18:55:12.072637 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.072598    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:12.120001 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.119959    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e81e058-5962-4007-a197-7407e64604c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-856efad4-llmpt\" (UID: \"0e81e058-5962-4007-a197-7407e64604c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"
Oct 30 18:55:12.120235 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.120028    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs2n7\" (UniqueName: \"kubernetes.io/projected/0e81e058-5962-4007-a197-7407e64604c7-kube-api-access-zs2n7\") pod \"maas-api-revoke-sub-856efad4-llmpt\" (UID: \"0e81e058-5962-4007-a197-7407e64604c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"
Oct 30 18:55:12.120235 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.120110    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e81e058-5962-4007-a197-7407e64604c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-856efad4-llmpt\" (UID: \"0e81e058-5962-4007-a197-7407e64604c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"
Oct 30 18:55:12.121958 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.121936    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:55:12.124302 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.124283    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"]
Oct 30 18:55:12.221087 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.221051    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e81e058-5962-4007-a197-7407e64604c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-856efad4-llmpt\" (UID: \"0e81e058-5962-4007-a197-7407e64604c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"
Oct 30 18:55:12.221318 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.221132    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs2n7\" (UniqueName: \"kubernetes.io/projected/0e81e058-5962-4007-a197-7407e64604c7-kube-api-access-zs2n7\") pod \"maas-api-revoke-sub-856efad4-llmpt\" (UID: \"0e81e058-5962-4007-a197-7407e64604c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"
Oct 30 18:55:12.221318 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.221213    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e81e058-5962-4007-a197-7407e64604c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-856efad4-llmpt\" (UID: \"0e81e058-5962-4007-a197-7407e64604c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"
Oct 30 18:55:12.221918 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.221893    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e81e058-5962-4007-a197-7407e64604c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-856efad4-llmpt\" (UID: \"0e81e058-5962-4007-a197-7407e64604c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"
Oct 30 18:55:12.223841 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.223809    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e81e058-5962-4007-a197-7407e64604c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-856efad4-llmpt\" (UID: \"0e81e058-5962-4007-a197-7407e64604c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"
Oct 30 18:55:12.278926 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.278843    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs2n7\" (UniqueName: \"kubernetes.io/projected/0e81e058-5962-4007-a197-7407e64604c7-kube-api-access-zs2n7\") pod \"maas-api-revoke-sub-856efad4-llmpt\" (UID: \"0e81e058-5962-4007-a197-7407e64604c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"
Oct 30 18:55:12.334478 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.334435    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"
Oct 30 18:55:12.711321 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:12.711295    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"]
Oct 30 18:55:12.714497 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:55:12.714473    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e81e058_5962_4007_a197_7407e64604c7.slice/crio-e50602b36e424e28968fd7cd8de6bba88d759cee2f46efa861e9cc5e2599cdd8 WatchSource:0}: Error finding container e50602b36e424e28968fd7cd8de6bba88d759cee2f46efa861e9cc5e2599cdd8: Status 404 returned error can't find the container with id e50602b36e424e28968fd7cd8de6bba88d759cee2f46efa861e9cc5e2599cdd8
Oct 30 18:55:13.442864 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:13.442817    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt" event={"ID":"0e81e058-5962-4007-a197-7407e64604c7","Type":"ContainerStarted","Data":"096a6655a1511016ad6de3457aab2f0b405c97a9626130c51b4dbb594fe41b1c"}
Oct 30 18:55:13.443325 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:13.442882    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt" event={"ID":"0e81e058-5962-4007-a197-7407e64604c7","Type":"ContainerStarted","Data":"e50602b36e424e28968fd7cd8de6bba88d759cee2f46efa861e9cc5e2599cdd8"}
Oct 30 18:55:13.559934 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:13.559875    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt" podStartSLOduration=2.559854378 podStartE2EDuration="2.559854378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:13.55835075 +0000 UTC m=+952.201518366" watchObservedRunningTime="2026-10-30 18:55:13.559854378 +0000 UTC m=+952.203021993"
Oct 30 18:55:14.450874 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:14.450838    6014 generic.go:356] "Generic (PLEG): container finished" podID="0e81e058-5962-4007-a197-7407e64604c7" containerID="096a6655a1511016ad6de3457aab2f0b405c97a9626130c51b4dbb594fe41b1c" exitCode=0
Oct 30 18:55:14.451397 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:14.450932    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt" event={"ID":"0e81e058-5962-4007-a197-7407e64604c7","Type":"ContainerDied","Data":"096a6655a1511016ad6de3457aab2f0b405c97a9626130c51b4dbb594fe41b1c"}
Oct 30 18:55:15.422799 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:15.422760    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:55:15.601407 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:15.601378    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"
Oct 30 18:55:15.661746 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:15.661705    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e81e058-5962-4007-a197-7407e64604c7-kube-api-access-zs2n7\" (UniqueName: \"kubernetes.io/projected/0e81e058-5962-4007-a197-7407e64604c7-kube-api-access-zs2n7\") pod \"0e81e058-5962-4007-a197-7407e64604c7\" (UID: \"0e81e058-5962-4007-a197-7407e64604c7\") "
Oct 30 18:55:15.662001 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:15.661951    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e81e058-5962-4007-a197-7407e64604c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e81e058-5962-4007-a197-7407e64604c7-maas-api-xks-ca\") pod \"0e81e058-5962-4007-a197-7407e64604c7\" (UID: \"0e81e058-5962-4007-a197-7407e64604c7\") "
Oct 30 18:55:15.662001 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:15.661984    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e81e058-5962-4007-a197-7407e64604c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e81e058-5962-4007-a197-7407e64604c7-maas-api-openshift-ca\") pod \"0e81e058-5962-4007-a197-7407e64604c7\" (UID: \"0e81e058-5962-4007-a197-7407e64604c7\") "
Oct 30 18:55:15.662436 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:15.662408    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e81e058-5962-4007-a197-7407e64604c7-maas-api-openshift-ca" pod "0e81e058-5962-4007-a197-7407e64604c7" (UID: "0e81e058-5962-4007-a197-7407e64604c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:15.662711 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:15.662554    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e81e058-5962-4007-a197-7407e64604c7-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:15.664009 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:15.663983    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e81e058-5962-4007-a197-7407e64604c7-kube-api-access-zs2n7" pod "0e81e058-5962-4007-a197-7407e64604c7" (UID: "0e81e058-5962-4007-a197-7407e64604c7"). InnerVolumeSpecName "kube-api-access-zs2n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:15.664096 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:15.664026    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e81e058-5962-4007-a197-7407e64604c7-maas-api-xks-ca" pod "0e81e058-5962-4007-a197-7407e64604c7" (UID: "0e81e058-5962-4007-a197-7407e64604c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:15.763788 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:15.763713    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e81e058-5962-4007-a197-7407e64604c7-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:15.763788 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:15.763742    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs2n7\" (UniqueName: \"kubernetes.io/projected/0e81e058-5962-4007-a197-7407e64604c7-kube-api-access-zs2n7\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:16.467951 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:16.467916    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt" event={"ID":"0e81e058-5962-4007-a197-7407e64604c7","Type":"ContainerDied","Data":"e50602b36e424e28968fd7cd8de6bba88d759cee2f46efa861e9cc5e2599cdd8"}
Oct 30 18:55:16.467951 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:16.467953    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"
Oct 30 18:55:16.468259 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:16.467966    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e50602b36e424e28968fd7cd8de6bba88d759cee2f46efa861e9cc5e2599cdd8"
Oct 30 18:55:23.258673 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:55:23.258628    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:55:27.269669 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.269632    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"]
Oct 30 18:55:27.270234 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.270220    6014 cpu_manager.go:414] "containerMap: removing container" podUID="0e81e058-5962-4007-a197-7407e64604c7" containerName="revoke-keys"
Oct 30 18:55:27.270290 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.270235    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e81e058-5962-4007-a197-7407e64604c7" containerName="revoke-keys"
Oct 30 18:55:27.270341 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.270331    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e81e058-5962-4007-a197-7407e64604c7" containerName="revoke-keys"
Oct 30 18:55:27.273917 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.273900    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"
Oct 30 18:55:27.376967 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.376935    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:27.377175 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.376978    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:55:27.380085 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.380064    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"]
Oct 30 18:55:27.408743 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.408712    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flhcg\" (UniqueName: \"kubernetes.io/projected/1024980a-606c-4968-a5f0-475a3cc63b92-kube-api-access-flhcg\") pod \"maas-api-revoke-sub-5450e8d4-krbf5\" (UID: \"1024980a-606c-4968-a5f0-475a3cc63b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"
Oct 30 18:55:27.408885 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.408770    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5450e8d4-krbf5\" (UID: \"1024980a-606c-4968-a5f0-475a3cc63b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"
Oct 30 18:55:27.408933 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.408903    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5450e8d4-krbf5\" (UID: \"1024980a-606c-4968-a5f0-475a3cc63b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"
Oct 30 18:55:27.510179 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.510128    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5450e8d4-krbf5\" (UID: \"1024980a-606c-4968-a5f0-475a3cc63b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"
Oct 30 18:55:27.510376 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.510276    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flhcg\" (UniqueName: \"kubernetes.io/projected/1024980a-606c-4968-a5f0-475a3cc63b92-kube-api-access-flhcg\") pod \"maas-api-revoke-sub-5450e8d4-krbf5\" (UID: \"1024980a-606c-4968-a5f0-475a3cc63b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"
Oct 30 18:55:27.510376 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.510338    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5450e8d4-krbf5\" (UID: \"1024980a-606c-4968-a5f0-475a3cc63b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"
Oct 30 18:55:27.510945 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.510924    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5450e8d4-krbf5\" (UID: \"1024980a-606c-4968-a5f0-475a3cc63b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"
Oct 30 18:55:27.512608 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.512589    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5450e8d4-krbf5\" (UID: \"1024980a-606c-4968-a5f0-475a3cc63b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"
Oct 30 18:55:27.567782 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.567743    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flhcg\" (UniqueName: \"kubernetes.io/projected/1024980a-606c-4968-a5f0-475a3cc63b92-kube-api-access-flhcg\") pod \"maas-api-revoke-sub-5450e8d4-krbf5\" (UID: \"1024980a-606c-4968-a5f0-475a3cc63b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"
Oct 30 18:55:27.595405 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.595373    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"
Oct 30 18:55:27.973889 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:27.973861    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"]
Oct 30 18:55:27.976367 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:55:27.976330    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1024980a_606c_4968_a5f0_475a3cc63b92.slice/crio-215c29c9965a112b76ec99cd243baebf3674a509037a49b77a4c58af42f2dd00 WatchSource:0}: Error finding container 215c29c9965a112b76ec99cd243baebf3674a509037a49b77a4c58af42f2dd00: Status 404 returned error can't find the container with id 215c29c9965a112b76ec99cd243baebf3674a509037a49b77a4c58af42f2dd00
Oct 30 18:55:28.538300 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:28.538215    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5" event={"ID":"1024980a-606c-4968-a5f0-475a3cc63b92","Type":"ContainerStarted","Data":"14dde87cce76c56a5e641cfeb26114b48b83b813f7056f6363ec739875469f89"}
Oct 30 18:55:28.538300 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:28.538275    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5" event={"ID":"1024980a-606c-4968-a5f0-475a3cc63b92","Type":"ContainerStarted","Data":"215c29c9965a112b76ec99cd243baebf3674a509037a49b77a4c58af42f2dd00"}
Oct 30 18:55:28.651915 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:28.651854    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5" podStartSLOduration=1.651834408 podStartE2EDuration="1.651834408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:28.651380136 +0000 UTC m=+967.294547760" watchObservedRunningTime="2026-10-30 18:55:28.651834408 +0000 UTC m=+967.295002023"
Oct 30 18:55:29.545585 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:29.545549    6014 generic.go:356] "Generic (PLEG): container finished" podID="1024980a-606c-4968-a5f0-475a3cc63b92" containerID="14dde87cce76c56a5e641cfeb26114b48b83b813f7056f6363ec739875469f89" exitCode=0
Oct 30 18:55:29.546008 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:29.545631    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5" event={"ID":"1024980a-606c-4968-a5f0-475a3cc63b92","Type":"ContainerDied","Data":"14dde87cce76c56a5e641cfeb26114b48b83b813f7056f6363ec739875469f89"}
Oct 30 18:55:30.709833 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:30.709809    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"
Oct 30 18:55:30.751633 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:30.751596    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-openshift-ca\") pod \"1024980a-606c-4968-a5f0-475a3cc63b92\" (UID: \"1024980a-606c-4968-a5f0-475a3cc63b92\") "
Oct 30 18:55:30.751840 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:30.751756    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-xks-ca\") pod \"1024980a-606c-4968-a5f0-475a3cc63b92\" (UID: \"1024980a-606c-4968-a5f0-475a3cc63b92\") "
Oct 30 18:55:30.751840 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:30.751793    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1024980a-606c-4968-a5f0-475a3cc63b92-kube-api-access-flhcg\" (UniqueName: \"kubernetes.io/projected/1024980a-606c-4968-a5f0-475a3cc63b92-kube-api-access-flhcg\") pod \"1024980a-606c-4968-a5f0-475a3cc63b92\" (UID: \"1024980a-606c-4968-a5f0-475a3cc63b92\") "
Oct 30 18:55:30.752040 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:30.752017    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-openshift-ca" pod "1024980a-606c-4968-a5f0-475a3cc63b92" (UID: "1024980a-606c-4968-a5f0-475a3cc63b92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:30.752276 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:30.752259    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:30.754025 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:30.754001    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-xks-ca" pod "1024980a-606c-4968-a5f0-475a3cc63b92" (UID: "1024980a-606c-4968-a5f0-475a3cc63b92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:30.754148 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:30.754116    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1024980a-606c-4968-a5f0-475a3cc63b92-kube-api-access-flhcg" pod "1024980a-606c-4968-a5f0-475a3cc63b92" (UID: "1024980a-606c-4968-a5f0-475a3cc63b92"). InnerVolumeSpecName "kube-api-access-flhcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:30.853042 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:30.852994    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1024980a-606c-4968-a5f0-475a3cc63b92-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:30.853042 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:30.853030    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flhcg\" (UniqueName: \"kubernetes.io/projected/1024980a-606c-4968-a5f0-475a3cc63b92-kube-api-access-flhcg\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:31.560116 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:31.560081    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5" event={"ID":"1024980a-606c-4968-a5f0-475a3cc63b92","Type":"ContainerDied","Data":"215c29c9965a112b76ec99cd243baebf3674a509037a49b77a4c58af42f2dd00"}
Oct 30 18:55:31.560116 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:31.560103    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"
Oct 30 18:55:31.560332 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:31.560175    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="215c29c9965a112b76ec99cd243baebf3674a509037a49b77a4c58af42f2dd00"
Oct 30 18:55:36.786841 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.786792    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"]
Oct 30 18:55:36.787733 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.787713    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1024980a-606c-4968-a5f0-475a3cc63b92" containerName="revoke-keys"
Oct 30 18:55:36.787820 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.787734    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1024980a-606c-4968-a5f0-475a3cc63b92" containerName="revoke-keys"
Oct 30 18:55:36.787856 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.787837    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1024980a-606c-4968-a5f0-475a3cc63b92" containerName="revoke-keys"
Oct 30 18:55:36.790174 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.790129    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.835929 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.835890    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tls-certs\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.836123 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.835953    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-dshm\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.836123 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.836041    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-home\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.836279 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.836173    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-model-cache\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.836279 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.836238    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tmp-dir\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.836386 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.836315    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd7p7\" (UniqueName: \"kubernetes.io/projected/41ac9daa-792e-4206-baa8-db8f3c4ccce9-kube-api-access-wd7p7\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.894405 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.894372    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f5763a\"/\"openshift-service-ca.crt\""
Oct 30 18:55:36.894405 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.894374    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f5763a\"/\"kube-root-ca.crt\""
Oct 30 18:55:36.894629 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.894538    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f5763a\"/\"default-dockercfg-64n6t\""
Oct 30 18:55:36.894629 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.894554    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f5763a\"/\"premium-w0-f5763a-kserve-self-signed-certs\""
Oct 30 18:55:36.896715 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.896695    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"]
Oct 30 18:55:36.937824 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.937786    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tls-certs\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.938018 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.937842    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-dshm\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.938018 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.937888    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-home\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.938018 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.937972    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-model-cache\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.938252 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.938037    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tmp-dir\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.938252 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.938060    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd7p7\" (UniqueName: \"kubernetes.io/projected/41ac9daa-792e-4206-baa8-db8f3c4ccce9-kube-api-access-wd7p7\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.938532 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.938502    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tmp-dir\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.938753 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.938687    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-model-cache\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.938956 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.938911    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-home\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.941198 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.941128    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tls-certs\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.958312 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.958280    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-dshm\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:36.999734 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:36.999697    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd7p7\" (UniqueName: \"kubernetes.io/projected/41ac9daa-792e-4206-baa8-db8f3c4ccce9-kube-api-access-wd7p7\") pod \"premium-w0-f5763a-kserve-ccb895d7d-6xnqs\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") " pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:37.101735 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:37.101686    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:37.478488 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:37.478465    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"]
Oct 30 18:55:37.480736 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:55:37.480705    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41ac9daa_792e_4206_baa8_db8f3c4ccce9.slice/crio-983b6c4bfeea13d821bcee5de01f13d8a99f62e51c77e61f9120a4e4c864bd5c WatchSource:0}: Error finding container 983b6c4bfeea13d821bcee5de01f13d8a99f62e51c77e61f9120a4e4c864bd5c: Status 404 returned error can't find the container with id 983b6c4bfeea13d821bcee5de01f13d8a99f62e51c77e61f9120a4e4c864bd5c
Oct 30 18:55:37.597867 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:37.597824    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" event={"ID":"41ac9daa-792e-4206-baa8-db8f3c4ccce9","Type":"ContainerStarted","Data":"d5ff7ef6c10785f135b116c16e9d26a8e26b41a576d9232a16aaadc5f03f9ab1"}
Oct 30 18:55:37.598050 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:37.597880    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" event={"ID":"41ac9daa-792e-4206-baa8-db8f3c4ccce9","Type":"ContainerStarted","Data":"983b6c4bfeea13d821bcee5de01f13d8a99f62e51c77e61f9120a4e4c864bd5c"}
Oct 30 18:55:37.712719 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:37.712660    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podStartSLOduration=1.712644585 podStartE2EDuration="1.712644585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:37.712016172 +0000 UTC m=+976.355183786" watchObservedRunningTime="2026-10-30 18:55:37.712644585 +0000 UTC m=+976.355812199"
Oct 30 18:55:38.102682 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:38.102644    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:39.575709 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.575679    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"]
Oct 30 18:55:39.578804 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.578785    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"
Oct 30 18:55:39.631223 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.631187    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:39.669354 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.669300    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr42g\" (UniqueName: \"kubernetes.io/projected/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-kube-api-access-pr42g\") pod \"maas-api-revoke-sub-0a880054-tns4d\" (UID: \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"
Oct 30 18:55:39.669556 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.669431    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a880054-tns4d\" (UID: \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"
Oct 30 18:55:39.669634 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.669623    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a880054-tns4d\" (UID: \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"
Oct 30 18:55:39.680732 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.680707    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:55:39.683214 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.683189    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"]
Oct 30 18:55:39.770811 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.770771    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr42g\" (UniqueName: \"kubernetes.io/projected/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-kube-api-access-pr42g\") pod \"maas-api-revoke-sub-0a880054-tns4d\" (UID: \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"
Oct 30 18:55:39.770989 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.770828    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a880054-tns4d\" (UID: \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"
Oct 30 18:55:39.770989 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.770949    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a880054-tns4d\" (UID: \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"
Oct 30 18:55:39.771642 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.771617    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a880054-tns4d\" (UID: \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"
Oct 30 18:55:39.773396 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.773373    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a880054-tns4d\" (UID: \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"
Oct 30 18:55:39.829386 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.829313    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr42g\" (UniqueName: \"kubernetes.io/projected/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-kube-api-access-pr42g\") pod \"maas-api-revoke-sub-0a880054-tns4d\" (UID: \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"
Oct 30 18:55:39.892857 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:39.892826    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"
Oct 30 18:55:40.291064 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:40.290977    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"]
Oct 30 18:55:40.299375 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:55:40.299343    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded200bf7_d482_4ec9_b125_84c9eeea2c4e.slice/crio-7f77bcf5f32abf0f842cf0376c3ae4b3a58bedb71f27b478eb19fa71f3ea7a10 WatchSource:0}: Error finding container 7f77bcf5f32abf0f842cf0376c3ae4b3a58bedb71f27b478eb19fa71f3ea7a10: Status 404 returned error can't find the container with id 7f77bcf5f32abf0f842cf0376c3ae4b3a58bedb71f27b478eb19fa71f3ea7a10
Oct 30 18:55:40.619168 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:40.619117    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d" event={"ID":"ed200bf7-d482-4ec9-b125-84c9eeea2c4e","Type":"ContainerStarted","Data":"047d67fd961295dd3c2f7fdd74cea0410a0393f8bb86df40b6376cff6b954113"}
Oct 30 18:55:40.619575 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:40.619195    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d" event={"ID":"ed200bf7-d482-4ec9-b125-84c9eeea2c4e","Type":"ContainerStarted","Data":"7f77bcf5f32abf0f842cf0376c3ae4b3a58bedb71f27b478eb19fa71f3ea7a10"}
Oct 30 18:55:40.735247 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:40.735192    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d" podStartSLOduration=1.735177083 podStartE2EDuration="1.735177083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:40.732693096 +0000 UTC m=+979.375860710" watchObservedRunningTime="2026-10-30 18:55:40.735177083 +0000 UTC m=+979.378344696"
Oct 30 18:55:41.627358 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:41.627321    6014 generic.go:356] "Generic (PLEG): container finished" podID="ed200bf7-d482-4ec9-b125-84c9eeea2c4e" containerID="047d67fd961295dd3c2f7fdd74cea0410a0393f8bb86df40b6376cff6b954113" exitCode=0
Oct 30 18:55:41.627762 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:41.627399    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d" event={"ID":"ed200bf7-d482-4ec9-b125-84c9eeea2c4e","Type":"ContainerDied","Data":"047d67fd961295dd3c2f7fdd74cea0410a0393f8bb86df40b6376cff6b954113"}
Oct 30 18:55:42.778019 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:42.777993    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"
Oct 30 18:55:42.906633 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:42.906540    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-xks-ca\") pod \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\" (UID: \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\") "
Oct 30 18:55:42.906633 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:42.906585    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-openshift-ca\") pod \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\" (UID: \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\") "
Oct 30 18:55:42.906892 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:42.906709    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-kube-api-access-pr42g\" (UniqueName: \"kubernetes.io/projected/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-kube-api-access-pr42g\") pod \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\" (UID: \"ed200bf7-d482-4ec9-b125-84c9eeea2c4e\") "
Oct 30 18:55:42.907080 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:42.907046    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-openshift-ca" pod "ed200bf7-d482-4ec9-b125-84c9eeea2c4e" (UID: "ed200bf7-d482-4ec9-b125-84c9eeea2c4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:42.907287 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:42.907261    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:42.909035 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:42.909008    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-xks-ca" pod "ed200bf7-d482-4ec9-b125-84c9eeea2c4e" (UID: "ed200bf7-d482-4ec9-b125-84c9eeea2c4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:42.909122 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:42.909063    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-kube-api-access-pr42g" pod "ed200bf7-d482-4ec9-b125-84c9eeea2c4e" (UID: "ed200bf7-d482-4ec9-b125-84c9eeea2c4e"). InnerVolumeSpecName "kube-api-access-pr42g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:43.008694 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:43.008659    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:43.008694 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:43.008689    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr42g\" (UniqueName: \"kubernetes.io/projected/ed200bf7-d482-4ec9-b125-84c9eeea2c4e-kube-api-access-pr42g\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:43.642417 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:43.642389    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"
Oct 30 18:55:43.642417 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:43.642403    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d" event={"ID":"ed200bf7-d482-4ec9-b125-84c9eeea2c4e","Type":"ContainerDied","Data":"7f77bcf5f32abf0f842cf0376c3ae4b3a58bedb71f27b478eb19fa71f3ea7a10"}
Oct 30 18:55:43.642622 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:43.642455    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f77bcf5f32abf0f842cf0376c3ae4b3a58bedb71f27b478eb19fa71f3ea7a10"
Oct 30 18:55:46.271628 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.271595    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"]
Oct 30 18:55:46.273271 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.273240    6014 cpu_manager.go:414] "containerMap: removing container" podUID="ed200bf7-d482-4ec9-b125-84c9eeea2c4e" containerName="revoke-keys"
Oct 30 18:55:46.273271 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.273270    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed200bf7-d482-4ec9-b125-84c9eeea2c4e" containerName="revoke-keys"
Oct 30 18:55:46.273626 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.273588    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed200bf7-d482-4ec9-b125-84c9eeea2c4e" containerName="revoke-keys"
Oct 30 18:55:46.278506 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.278482    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.330894 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.330862    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-1290a6-openshift-default-dockercfg-zvdjm\""
Oct 30 18:55:46.345600 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.345563    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.345748 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.345644    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.345748 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.345685    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.345822 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.345802    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdmg\" (UniqueName: \"kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-kube-api-access-9hdmg\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.345960 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.345946    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.345998 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.345970    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.345998 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.345995    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.346065 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.346015    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.346121 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.346099    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.384691 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.384660    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"]
Oct 30 18:55:46.447222 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.447182    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdmg\" (UniqueName: \"kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-kube-api-access-9hdmg\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.447405 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.447318    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.447405 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.447343    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.447405 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.447368    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.447405 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.447386    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.447612 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.447407    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.447612 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.447433    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.447612 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.447514    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.447612 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.447539    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.447860 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.447821    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.447860 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.447851    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.447982 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.447949    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.448061 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.448036    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.448247 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.448230    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.449806 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.449781    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.450026 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.450010    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.504641 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.504600    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.504799 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.504651    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdmg\" (UniqueName: \"kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-kube-api-access-9hdmg\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:46.593282 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:46.593253    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:47.021093 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:47.021010    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"]
Oct 30 18:55:47.061649 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:55:47.061619    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4970c31f_3fb0_4265_8e83_94c3f0fa1b99.slice/crio-f8a958b22eb7914c8b4dd2174d2b214eff9a810b2c99c7fee9b42ec699e31857 WatchSource:0}: Error finding container f8a958b22eb7914c8b4dd2174d2b214eff9a810b2c99c7fee9b42ec699e31857: Status 404 returned error can't find the container with id f8a958b22eb7914c8b4dd2174d2b214eff9a810b2c99c7fee9b42ec699e31857
Oct 30 18:55:47.064049 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:47.064013    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:55:47.064162 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:47.064085    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:55:47.064162 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:47.064112    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:55:47.102077 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:47.102050    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:47.116916 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:47.116888    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:47.680016 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:47.679979    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx" event={"ID":"4970c31f-3fb0-4265-8e83-94c3f0fa1b99","Type":"ContainerStarted","Data":"eefe9595ecbfab0ed00d68dba353073200e58d77bb47d84acb25ecac3d8fd08c"}
Oct 30 18:55:47.680506 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:47.680032    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx" event={"ID":"4970c31f-3fb0-4265-8e83-94c3f0fa1b99","Type":"ContainerStarted","Data":"f8a958b22eb7914c8b4dd2174d2b214eff9a810b2c99c7fee9b42ec699e31857"}
Oct 30 18:55:47.692297 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:47.692267    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 18:55:47.730724 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:47.730691    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:55:47.803132 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:47.803077    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx" podStartSLOduration=1.803063026 podStartE2EDuration="1.803063026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:47.800025655 +0000 UTC m=+986.443193274" watchObservedRunningTime="2026-10-30 18:55:47.803063026 +0000 UTC m=+986.446230639"
Oct 30 18:55:48.596538 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:48.596478    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:48.600163 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:48.598510    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 18:55:48.600163 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:48.598582    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx" podUID="4970c31f-3fb0-4265-8e83-94c3f0fa1b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 18:55:49.594576 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:49.594540    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 18:55:49.595023 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:49.594592    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx" podUID="4970c31f-3fb0-4265-8e83-94c3f0fa1b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 18:55:50.593758 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:50.593726    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 18:55:50.593947 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:50.593781    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx" podUID="4970c31f-3fb0-4265-8e83-94c3f0fa1b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 18:55:51.594589 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:51.594554    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 18:55:51.595074 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:51.594614    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx" podUID="4970c31f-3fb0-4265-8e83-94c3f0fa1b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 18:55:52.594618 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:52.594583    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 18:55:52.595124 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:52.594646    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx" podUID="4970c31f-3fb0-4265-8e83-94c3f0fa1b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 18:55:53.258083 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:55:53.258048    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:55:53.593766 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:53.593728    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 18:55:53.594043 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:53.593794    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx" podUID="4970c31f-3fb0-4265-8e83-94c3f0fa1b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 18:55:54.598720 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:54.598690    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:54.599117 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:54.599092    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:55:54.600037 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:55:54.600014    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 18:56:01.347409 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.347324    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"]
Oct 30 18:56:01.352898 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.352881    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"
Oct 30 18:56:01.386517 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.386478    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrl99\" (UniqueName: \"kubernetes.io/projected/489fa858-be00-432c-959d-b48f4e0d2ee4-kube-api-access-hrl99\") pod \"maas-api-revoke-sub-0981c105-9q92x\" (UID: \"489fa858-be00-432c-959d-b48f4e0d2ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"
Oct 30 18:56:01.386673 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.386627    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0981c105-9q92x\" (UID: \"489fa858-be00-432c-959d-b48f4e0d2ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"
Oct 30 18:56:01.386748 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.386701    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0981c105-9q92x\" (UID: \"489fa858-be00-432c-959d-b48f4e0d2ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"
Oct 30 18:56:01.408388 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.408358    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"]
Oct 30 18:56:01.454381 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.454352    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:01.454546 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.454435    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:56:01.488261 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.488233    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrl99\" (UniqueName: \"kubernetes.io/projected/489fa858-be00-432c-959d-b48f4e0d2ee4-kube-api-access-hrl99\") pod \"maas-api-revoke-sub-0981c105-9q92x\" (UID: \"489fa858-be00-432c-959d-b48f4e0d2ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"
Oct 30 18:56:01.488468 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.488339    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0981c105-9q92x\" (UID: \"489fa858-be00-432c-959d-b48f4e0d2ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"
Oct 30 18:56:01.488468 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.488399    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0981c105-9q92x\" (UID: \"489fa858-be00-432c-959d-b48f4e0d2ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"
Oct 30 18:56:01.489281 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.489251    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0981c105-9q92x\" (UID: \"489fa858-be00-432c-959d-b48f4e0d2ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"
Oct 30 18:56:01.490836 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.490816    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0981c105-9q92x\" (UID: \"489fa858-be00-432c-959d-b48f4e0d2ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"
Oct 30 18:56:01.545504 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.545476    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrl99\" (UniqueName: \"kubernetes.io/projected/489fa858-be00-432c-959d-b48f4e0d2ee4-kube-api-access-hrl99\") pod \"maas-api-revoke-sub-0981c105-9q92x\" (UID: \"489fa858-be00-432c-959d-b48f4e0d2ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"
Oct 30 18:56:01.667065 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:01.666975    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"
Oct 30 18:56:02.046988 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:02.046960    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"]
Oct 30 18:56:02.049210 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:56:02.049178    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489fa858_be00_432c_959d_b48f4e0d2ee4.slice/crio-3eaf462a299f74ec3594f52c332f384acb8d80e4255bf3643dbd54e85b26295f WatchSource:0}: Error finding container 3eaf462a299f74ec3594f52c332f384acb8d80e4255bf3643dbd54e85b26295f: Status 404 returned error can't find the container with id 3eaf462a299f74ec3594f52c332f384acb8d80e4255bf3643dbd54e85b26295f
Oct 30 18:56:02.785873 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:02.785768    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x" event={"ID":"489fa858-be00-432c-959d-b48f4e0d2ee4","Type":"ContainerStarted","Data":"13d399c2a57de5dbfa31975945971d837c4ca7c666129062307b3d5a7921856d"}
Oct 30 18:56:02.785873 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:02.785833    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x" event={"ID":"489fa858-be00-432c-959d-b48f4e0d2ee4","Type":"ContainerStarted","Data":"3eaf462a299f74ec3594f52c332f384acb8d80e4255bf3643dbd54e85b26295f"}
Oct 30 18:56:03.795812 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:03.795772    6014 generic.go:356] "Generic (PLEG): container finished" podID="489fa858-be00-432c-959d-b48f4e0d2ee4" containerID="13d399c2a57de5dbfa31975945971d837c4ca7c666129062307b3d5a7921856d" exitCode=0
Oct 30 18:56:03.796414 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:03.795868    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x" event={"ID":"489fa858-be00-432c-959d-b48f4e0d2ee4","Type":"ContainerDied","Data":"13d399c2a57de5dbfa31975945971d837c4ca7c666129062307b3d5a7921856d"}
Oct 30 18:56:04.961083 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:04.961059    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"
Oct 30 18:56:05.035675 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:05.035642    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-xks-ca\") pod \"489fa858-be00-432c-959d-b48f4e0d2ee4\" (UID: \"489fa858-be00-432c-959d-b48f4e0d2ee4\") "
Oct 30 18:56:05.035900 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:05.035689    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489fa858-be00-432c-959d-b48f4e0d2ee4-kube-api-access-hrl99\" (UniqueName: \"kubernetes.io/projected/489fa858-be00-432c-959d-b48f4e0d2ee4-kube-api-access-hrl99\") pod \"489fa858-be00-432c-959d-b48f4e0d2ee4\" (UID: \"489fa858-be00-432c-959d-b48f4e0d2ee4\") "
Oct 30 18:56:05.035900 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:05.035786    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-openshift-ca\") pod \"489fa858-be00-432c-959d-b48f4e0d2ee4\" (UID: \"489fa858-be00-432c-959d-b48f4e0d2ee4\") "
Oct 30 18:56:05.036193 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:05.036163    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-openshift-ca" pod "489fa858-be00-432c-959d-b48f4e0d2ee4" (UID: "489fa858-be00-432c-959d-b48f4e0d2ee4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:05.036335 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:05.036320    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:05.038001 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:05.037971    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-xks-ca" pod "489fa858-be00-432c-959d-b48f4e0d2ee4" (UID: "489fa858-be00-432c-959d-b48f4e0d2ee4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:05.038001 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:05.037981    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489fa858-be00-432c-959d-b48f4e0d2ee4-kube-api-access-hrl99" pod "489fa858-be00-432c-959d-b48f4e0d2ee4" (UID: "489fa858-be00-432c-959d-b48f4e0d2ee4"). InnerVolumeSpecName "kube-api-access-hrl99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:05.138104 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:05.138066    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489fa858-be00-432c-959d-b48f4e0d2ee4-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:05.138104 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:05.138099    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrl99\" (UniqueName: \"kubernetes.io/projected/489fa858-be00-432c-959d-b48f4e0d2ee4-kube-api-access-hrl99\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:05.809455 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:05.809369    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x" event={"ID":"489fa858-be00-432c-959d-b48f4e0d2ee4","Type":"ContainerDied","Data":"3eaf462a299f74ec3594f52c332f384acb8d80e4255bf3643dbd54e85b26295f"}
Oct 30 18:56:05.809455 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:05.809387    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"
Oct 30 18:56:05.809455 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:05.809416    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eaf462a299f74ec3594f52c332f384acb8d80e4255bf3643dbd54e85b26295f"
Oct 30 18:56:06.586790 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.586722    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"]
Oct 30 18:56:06.587555 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.587535    6014 cpu_manager.go:414] "containerMap: removing container" podUID="489fa858-be00-432c-959d-b48f4e0d2ee4" containerName="revoke-keys"
Oct 30 18:56:06.587555 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.587556    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489fa858-be00-432c-959d-b48f4e0d2ee4" containerName="revoke-keys"
Oct 30 18:56:06.587761 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.587747    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="489fa858-be00-432c-959d-b48f4e0d2ee4" containerName="revoke-keys"
Oct 30 18:56:06.591286 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.591261    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"
Oct 30 18:56:06.644433 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.644394    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:06.654579 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.654545    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpp5d\" (UniqueName: \"kubernetes.io/projected/cb979b30-1d50-4178-872f-3bb08acd3518-kube-api-access-kpp5d\") pod \"maas-api-revoke-sub-aff5cc44-ltwpk\" (UID: \"cb979b30-1d50-4178-872f-3bb08acd3518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"
Oct 30 18:56:06.654782 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.654749    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aff5cc44-ltwpk\" (UID: \"cb979b30-1d50-4178-872f-3bb08acd3518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"
Oct 30 18:56:06.654913 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.654895    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aff5cc44-ltwpk\" (UID: \"cb979b30-1d50-4178-872f-3bb08acd3518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"
Oct 30 18:56:06.693938 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.693906    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:56:06.696723 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.696697    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"]
Oct 30 18:56:06.756462 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.756427    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aff5cc44-ltwpk\" (UID: \"cb979b30-1d50-4178-872f-3bb08acd3518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"
Oct 30 18:56:06.756645 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.756484    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpp5d\" (UniqueName: \"kubernetes.io/projected/cb979b30-1d50-4178-872f-3bb08acd3518-kube-api-access-kpp5d\") pod \"maas-api-revoke-sub-aff5cc44-ltwpk\" (UID: \"cb979b30-1d50-4178-872f-3bb08acd3518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"
Oct 30 18:56:06.756645 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.756566    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aff5cc44-ltwpk\" (UID: \"cb979b30-1d50-4178-872f-3bb08acd3518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"
Oct 30 18:56:06.757199 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.757177    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aff5cc44-ltwpk\" (UID: \"cb979b30-1d50-4178-872f-3bb08acd3518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"
Oct 30 18:56:06.759215 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.759194    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aff5cc44-ltwpk\" (UID: \"cb979b30-1d50-4178-872f-3bb08acd3518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"
Oct 30 18:56:06.813589 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.813556    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpp5d\" (UniqueName: \"kubernetes.io/projected/cb979b30-1d50-4178-872f-3bb08acd3518-kube-api-access-kpp5d\") pod \"maas-api-revoke-sub-aff5cc44-ltwpk\" (UID: \"cb979b30-1d50-4178-872f-3bb08acd3518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"
Oct 30 18:56:06.906763 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:06.906661    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"
Oct 30 18:56:07.283294 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:07.283266    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"]
Oct 30 18:56:07.289292 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:56:07.289248    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb979b30_1d50_4178_872f_3bb08acd3518.slice/crio-db7780735bcdba0b5e8676f4a09a77e6ac98398807809d9eb4474a5c7489225f WatchSource:0}: Error finding container db7780735bcdba0b5e8676f4a09a77e6ac98398807809d9eb4474a5c7489225f: Status 404 returned error can't find the container with id db7780735bcdba0b5e8676f4a09a77e6ac98398807809d9eb4474a5c7489225f
Oct 30 18:56:07.822891 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:07.822856    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk" event={"ID":"cb979b30-1d50-4178-872f-3bb08acd3518","Type":"ContainerStarted","Data":"1f6dc833490352f76bc55866fcb063820003f98febc82321d5fc6fd29fe1cdf4"}
Oct 30 18:56:07.823424 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:07.822924    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk" event={"ID":"cb979b30-1d50-4178-872f-3bb08acd3518","Type":"ContainerStarted","Data":"db7780735bcdba0b5e8676f4a09a77e6ac98398807809d9eb4474a5c7489225f"}
Oct 30 18:56:07.936897 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:07.936843    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk" podStartSLOduration=1.9368295610000001 podStartE2EDuration="1.936829561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:07.935775289 +0000 UTC m=+1006.578942905" watchObservedRunningTime="2026-10-30 18:56:07.936829561 +0000 UTC m=+1006.579997176"
Oct 30 18:56:08.830146 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:08.830106    6014 generic.go:356] "Generic (PLEG): container finished" podID="cb979b30-1d50-4178-872f-3bb08acd3518" containerID="1f6dc833490352f76bc55866fcb063820003f98febc82321d5fc6fd29fe1cdf4" exitCode=0
Oct 30 18:56:08.830564 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:08.830203    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk" event={"ID":"cb979b30-1d50-4178-872f-3bb08acd3518","Type":"ContainerDied","Data":"1f6dc833490352f76bc55866fcb063820003f98febc82321d5fc6fd29fe1cdf4"}
Oct 30 18:56:09.986855 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:09.986828    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"
Oct 30 18:56:10.106342 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:10.106269    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb979b30-1d50-4178-872f-3bb08acd3518-kube-api-access-kpp5d\" (UniqueName: \"kubernetes.io/projected/cb979b30-1d50-4178-872f-3bb08acd3518-kube-api-access-kpp5d\") pod \"cb979b30-1d50-4178-872f-3bb08acd3518\" (UID: \"cb979b30-1d50-4178-872f-3bb08acd3518\") "
Oct 30 18:56:10.106491 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:10.106356    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-xks-ca\") pod \"cb979b30-1d50-4178-872f-3bb08acd3518\" (UID: \"cb979b30-1d50-4178-872f-3bb08acd3518\") "
Oct 30 18:56:10.106491 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:10.106383    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-openshift-ca\") pod \"cb979b30-1d50-4178-872f-3bb08acd3518\" (UID: \"cb979b30-1d50-4178-872f-3bb08acd3518\") "
Oct 30 18:56:10.106863 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:10.106833    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-openshift-ca" pod "cb979b30-1d50-4178-872f-3bb08acd3518" (UID: "cb979b30-1d50-4178-872f-3bb08acd3518"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:10.108633 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:10.108601    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb979b30-1d50-4178-872f-3bb08acd3518-kube-api-access-kpp5d" pod "cb979b30-1d50-4178-872f-3bb08acd3518" (UID: "cb979b30-1d50-4178-872f-3bb08acd3518"). InnerVolumeSpecName "kube-api-access-kpp5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:10.108741 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:10.108705    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-xks-ca" pod "cb979b30-1d50-4178-872f-3bb08acd3518" (UID: "cb979b30-1d50-4178-872f-3bb08acd3518"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:10.207750 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:10.207713    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:10.207750 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:10.207746    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb979b30-1d50-4178-872f-3bb08acd3518-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:10.207750 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:10.207756    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpp5d\" (UniqueName: \"kubernetes.io/projected/cb979b30-1d50-4178-872f-3bb08acd3518-kube-api-access-kpp5d\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:10.843712 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:10.843678    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk" event={"ID":"cb979b30-1d50-4178-872f-3bb08acd3518","Type":"ContainerDied","Data":"db7780735bcdba0b5e8676f4a09a77e6ac98398807809d9eb4474a5c7489225f"}
Oct 30 18:56:10.843908 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:10.843721    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"
Oct 30 18:56:10.843908 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:10.843729    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db7780735bcdba0b5e8676f4a09a77e6ac98398807809d9eb4474a5c7489225f"
Oct 30 18:56:14.047358 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.047323    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"]
Oct 30 18:56:14.048199 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.048177    6014 cpu_manager.go:414] "containerMap: removing container" podUID="cb979b30-1d50-4178-872f-3bb08acd3518" containerName="revoke-keys"
Oct 30 18:56:14.048199 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.048199    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb979b30-1d50-4178-872f-3bb08acd3518" containerName="revoke-keys"
Oct 30 18:56:14.048373 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.048356    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb979b30-1d50-4178-872f-3bb08acd3518" containerName="revoke-keys"
Oct 30 18:56:14.052000 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.051976    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"
Oct 30 18:56:14.106465 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.106415    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:14.107852 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.107834    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:56:14.153915 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.153875    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfbqg\" (UniqueName: \"kubernetes.io/projected/c05557c1-1499-4bec-bfa3-fbb27a93dec3-kube-api-access-jfbqg\") pod \"maas-api-revoke-sub-d9cfbce5-ndmdc\" (UID: \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"
Oct 30 18:56:14.154120 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.154099    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cfbce5-ndmdc\" (UID: \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"
Oct 30 18:56:14.154214 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.154134    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cfbce5-ndmdc\" (UID: \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"
Oct 30 18:56:14.160244 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.160216    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"]
Oct 30 18:56:14.254789 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.254754    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cfbce5-ndmdc\" (UID: \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"
Oct 30 18:56:14.254789 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.254791    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cfbce5-ndmdc\" (UID: \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"
Oct 30 18:56:14.255021 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.254857    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfbqg\" (UniqueName: \"kubernetes.io/projected/c05557c1-1499-4bec-bfa3-fbb27a93dec3-kube-api-access-jfbqg\") pod \"maas-api-revoke-sub-d9cfbce5-ndmdc\" (UID: \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"
Oct 30 18:56:14.255546 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.255523    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cfbce5-ndmdc\" (UID: \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"
Oct 30 18:56:14.257377 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.257355    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cfbce5-ndmdc\" (UID: \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"
Oct 30 18:56:14.312816 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.312744    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfbqg\" (UniqueName: \"kubernetes.io/projected/c05557c1-1499-4bec-bfa3-fbb27a93dec3-kube-api-access-jfbqg\") pod \"maas-api-revoke-sub-d9cfbce5-ndmdc\" (UID: \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"
Oct 30 18:56:14.371683 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.371646    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"
Oct 30 18:56:14.778995 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.778961    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"]
Oct 30 18:56:14.780832 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:56:14.780802    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05557c1_1499_4bec_bfa3_fbb27a93dec3.slice/crio-da0718a8b16bc8dec573e26f6ff8a3d5556e555672a3dba444058070b05a237d WatchSource:0}: Error finding container da0718a8b16bc8dec573e26f6ff8a3d5556e555672a3dba444058070b05a237d: Status 404 returned error can't find the container with id da0718a8b16bc8dec573e26f6ff8a3d5556e555672a3dba444058070b05a237d
Oct 30 18:56:14.870803 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:14.870767    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc" event={"ID":"c05557c1-1499-4bec-bfa3-fbb27a93dec3","Type":"ContainerStarted","Data":"da0718a8b16bc8dec573e26f6ff8a3d5556e555672a3dba444058070b05a237d"}
Oct 30 18:56:15.878291 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:15.878248    6014 generic.go:356] "Generic (PLEG): container finished" podID="c05557c1-1499-4bec-bfa3-fbb27a93dec3" containerID="f890389e400dc3dbdbeea176323208285433aa216ab779d54756615bb940b9ee" exitCode=0
Oct 30 18:56:15.878678 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:15.878381    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc" event={"ID":"c05557c1-1499-4bec-bfa3-fbb27a93dec3","Type":"ContainerDied","Data":"f890389e400dc3dbdbeea176323208285433aa216ab779d54756615bb940b9ee"}
Oct 30 18:56:17.041218 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:17.041194    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"
Oct 30 18:56:17.193550 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:17.193457    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05557c1-1499-4bec-bfa3-fbb27a93dec3-kube-api-access-jfbqg\" (UniqueName: \"kubernetes.io/projected/c05557c1-1499-4bec-bfa3-fbb27a93dec3-kube-api-access-jfbqg\") pod \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\" (UID: \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\") "
Oct 30 18:56:17.193740 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:17.193714    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-xks-ca\") pod \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\" (UID: \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\") "
Oct 30 18:56:17.193903 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:17.193832    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-openshift-ca\") pod \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\" (UID: \"c05557c1-1499-4bec-bfa3-fbb27a93dec3\") "
Oct 30 18:56:17.194308 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:17.194223    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-openshift-ca" pod "c05557c1-1499-4bec-bfa3-fbb27a93dec3" (UID: "c05557c1-1499-4bec-bfa3-fbb27a93dec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:17.194653 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:17.194632    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:17.195884 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:17.195860    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05557c1-1499-4bec-bfa3-fbb27a93dec3-kube-api-access-jfbqg" pod "c05557c1-1499-4bec-bfa3-fbb27a93dec3" (UID: "c05557c1-1499-4bec-bfa3-fbb27a93dec3"). InnerVolumeSpecName "kube-api-access-jfbqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:17.195978 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:17.195961    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-xks-ca" pod "c05557c1-1499-4bec-bfa3-fbb27a93dec3" (UID: "c05557c1-1499-4bec-bfa3-fbb27a93dec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:17.296131 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:17.296096    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05557c1-1499-4bec-bfa3-fbb27a93dec3-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:17.296131 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:17.296123    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfbqg\" (UniqueName: \"kubernetes.io/projected/c05557c1-1499-4bec-bfa3-fbb27a93dec3-kube-api-access-jfbqg\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:17.892747 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:17.892722    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"
Oct 30 18:56:17.892903 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:17.892721    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc" event={"ID":"c05557c1-1499-4bec-bfa3-fbb27a93dec3","Type":"ContainerDied","Data":"da0718a8b16bc8dec573e26f6ff8a3d5556e555672a3dba444058070b05a237d"}
Oct 30 18:56:17.892903 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:17.892878    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da0718a8b16bc8dec573e26f6ff8a3d5556e555672a3dba444058070b05a237d"
Oct 30 18:56:18.390153 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.390111    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"]
Oct 30 18:56:18.390993 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.390974    6014 cpu_manager.go:414] "containerMap: removing container" podUID="c05557c1-1499-4bec-bfa3-fbb27a93dec3" containerName="revoke-keys"
Oct 30 18:56:18.391080 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.390996    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05557c1-1499-4bec-bfa3-fbb27a93dec3" containerName="revoke-keys"
Oct 30 18:56:18.391229 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.391217    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05557c1-1499-4bec-bfa3-fbb27a93dec3" containerName="revoke-keys"
Oct 30 18:56:18.395271 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.395252    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.407978 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.407950    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.408108 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.407989    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.408230 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.408181    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.408287 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.408223    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qtbj\" (UniqueName: \"kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-kube-api-access-4qtbj\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.408345 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.408320    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.408401 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.408350    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.408480 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.408459    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.408524 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.408510    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.408599 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.408585    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.452592 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.452558    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bc98e1-openshift-default-dockercfg-mrm2g\""
Oct 30 18:56:18.509643 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.509606    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.509643 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.509648    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qtbj\" (UniqueName: \"kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-kube-api-access-4qtbj\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.509864 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.509687    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.509864 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.509830    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.509969 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.509916    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.510027 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.509980    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.510083 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.510028    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.510162 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.510084    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.510162 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.510110    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.510281 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.510243    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.510365 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.510326    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.510425 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.510399    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.510425 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.510409    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.510531 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.510450    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.512808 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.512780    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.512938 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.512782    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.517623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.517596    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"]
Oct 30 18:56:18.572781 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.572752    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.576988 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.576964    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qtbj\" (UniqueName: \"kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-kube-api-access-4qtbj\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:18.708973 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:18.708882    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:19.142275 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:19.142241    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"]
Oct 30 18:56:19.152085 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:56:19.152048    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ebfc68_3553_4bdb_9393_bc3b582c0bc0.slice/crio-a47695b7ae602d80419ff673c57ed0488b33313988354db0e470b6e2d4ca813a WatchSource:0}: Error finding container a47695b7ae602d80419ff673c57ed0488b33313988354db0e470b6e2d4ca813a: Status 404 returned error can't find the container with id a47695b7ae602d80419ff673c57ed0488b33313988354db0e470b6e2d4ca813a
Oct 30 18:56:19.154265 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:19.154221    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:19.154440 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:19.154301    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:19.154440 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:19.154335    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:19.906554 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:19.906513    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" event={"ID":"99ebfc68-3553-4bdb-9393-bc3b582c0bc0","Type":"ContainerStarted","Data":"f56562b375c2303dc0297bf5f0f3ac4a84fac7e52ed8a64731ca24513f9e29f6"}
Oct 30 18:56:19.907023 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:19.906579    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" event={"ID":"99ebfc68-3553-4bdb-9393-bc3b582c0bc0","Type":"ContainerStarted","Data":"a47695b7ae602d80419ff673c57ed0488b33313988354db0e470b6e2d4ca813a"}
Oct 30 18:56:20.025488 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:20.025426    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" podStartSLOduration=2.025410404 podStartE2EDuration="2.025410404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:20.024589114 +0000 UTC m=+1018.667756730" watchObservedRunningTime="2026-10-30 18:56:20.025410404 +0000 UTC m=+1018.668578018"
Oct 30 18:56:20.709219 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:20.709180    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:20.711298 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:20.711268    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:56:20.711424 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:20.711333    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" podUID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 18:56:21.709936 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:21.709905    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:56:21.710348 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:21.709958    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" podUID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 18:56:22.710169 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.710105    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:56:22.710639 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.710184    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" podUID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 18:56:22.808128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.808093    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"]
Oct 30 18:56:22.812686 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.812668    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"
Oct 30 18:56:22.883695 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.883654    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02fad9f6-g4dv4\" (UID: \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"
Oct 30 18:56:22.883879 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.883808    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02fad9f6-g4dv4\" (UID: \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"
Oct 30 18:56:22.883956 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.883934    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgq6d\" (UniqueName: \"kubernetes.io/projected/b0d15b24-c4d3-40a2-9203-9db14250fa7c-kube-api-access-zgq6d\") pod \"maas-api-revoke-sub-02fad9f6-g4dv4\" (UID: \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"
Oct 30 18:56:22.885442 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.885419    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:56:22.885587 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.885464    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:22.923548 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.923456    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"]
Oct 30 18:56:22.985827 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.985793    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgq6d\" (UniqueName: \"kubernetes.io/projected/b0d15b24-c4d3-40a2-9203-9db14250fa7c-kube-api-access-zgq6d\") pod \"maas-api-revoke-sub-02fad9f6-g4dv4\" (UID: \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"
Oct 30 18:56:22.985987 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.985909    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02fad9f6-g4dv4\" (UID: \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"
Oct 30 18:56:22.985987 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.985958    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02fad9f6-g4dv4\" (UID: \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"
Oct 30 18:56:22.986648 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.986624    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02fad9f6-g4dv4\" (UID: \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"
Oct 30 18:56:22.988517 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:22.988494    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02fad9f6-g4dv4\" (UID: \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"
Oct 30 18:56:23.042345 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:23.042314    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgq6d\" (UniqueName: \"kubernetes.io/projected/b0d15b24-c4d3-40a2-9203-9db14250fa7c-kube-api-access-zgq6d\") pod \"maas-api-revoke-sub-02fad9f6-g4dv4\" (UID: \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"
Oct 30 18:56:23.081286 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:23.081250    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:56:23.127027 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:23.126992    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"
Oct 30 18:56:23.303647 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:56:23.303591    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:56:23.566942 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:23.566904    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"]
Oct 30 18:56:23.709830 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:23.709793    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:56:23.709970 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:23.709869    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" podUID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 18:56:23.934636 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:23.934598    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4" event={"ID":"b0d15b24-c4d3-40a2-9203-9db14250fa7c","Type":"ContainerStarted","Data":"632e83eb97bc2025012f1265eb81357bd2edeaba1f3a40247a136de512837a5a"}
Oct 30 18:56:24.710171 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:24.710105    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:56:24.710490 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:24.710191    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" podUID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 18:56:24.944517 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:24.944483    6014 generic.go:356] "Generic (PLEG): container finished" podID="b0d15b24-c4d3-40a2-9203-9db14250fa7c" containerID="458b4bec26fa9173721518cd3d02b0385da3b1b36f40826aa5ccb5f5c2400bda" exitCode=0
Oct 30 18:56:24.944909 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:24.944531    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4" event={"ID":"b0d15b24-c4d3-40a2-9203-9db14250fa7c","Type":"ContainerDied","Data":"458b4bec26fa9173721518cd3d02b0385da3b1b36f40826aa5ccb5f5c2400bda"}
Oct 30 18:56:25.709671 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:25.709635    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:56:25.709888 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:25.709699    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" podUID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 18:56:26.103254 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.103228    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"
Oct 30 18:56:26.180366 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.180322    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-xks-ca\") pod \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\" (UID: \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\") "
Oct 30 18:56:26.180550 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.180414    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-openshift-ca\") pod \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\" (UID: \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\") "
Oct 30 18:56:26.180607 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.180596    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d15b24-c4d3-40a2-9203-9db14250fa7c-kube-api-access-zgq6d\" (UniqueName: \"kubernetes.io/projected/b0d15b24-c4d3-40a2-9203-9db14250fa7c-kube-api-access-zgq6d\") pod \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\" (UID: \"b0d15b24-c4d3-40a2-9203-9db14250fa7c\") "
Oct 30 18:56:26.180892 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.180857    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-openshift-ca" pod "b0d15b24-c4d3-40a2-9203-9db14250fa7c" (UID: "b0d15b24-c4d3-40a2-9203-9db14250fa7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:26.181290 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.181217    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:26.182793 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.182759    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d15b24-c4d3-40a2-9203-9db14250fa7c-kube-api-access-zgq6d" pod "b0d15b24-c4d3-40a2-9203-9db14250fa7c" (UID: "b0d15b24-c4d3-40a2-9203-9db14250fa7c"). InnerVolumeSpecName "kube-api-access-zgq6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:26.182916 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.182820    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-xks-ca" pod "b0d15b24-c4d3-40a2-9203-9db14250fa7c" (UID: "b0d15b24-c4d3-40a2-9203-9db14250fa7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:26.282570 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.282489    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d15b24-c4d3-40a2-9203-9db14250fa7c-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:26.282570 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.282521    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgq6d\" (UniqueName: \"kubernetes.io/projected/b0d15b24-c4d3-40a2-9203-9db14250fa7c-kube-api-access-zgq6d\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:26.714078 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.714046    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:26.714456 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.714431    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:26.715193 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.715175    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 18:56:26.824797 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.824768    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"]
Oct 30 18:56:26.825407 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.825392    6014 cpu_manager.go:414] "containerMap: removing container" podUID="b0d15b24-c4d3-40a2-9203-9db14250fa7c" containerName="revoke-keys"
Oct 30 18:56:26.825490 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.825408    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d15b24-c4d3-40a2-9203-9db14250fa7c" containerName="revoke-keys"
Oct 30 18:56:26.825537 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.825520    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d15b24-c4d3-40a2-9203-9db14250fa7c" containerName="revoke-keys"
Oct 30 18:56:26.828923 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.828906    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"
Oct 30 18:56:26.890822 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.890792    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1adf9edd-7w7ph\" (UID: \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"
Oct 30 18:56:26.891017 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.890989    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1adf9edd-7w7ph\" (UID: \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"
Oct 30 18:56:26.891151 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.891115    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hkbn\" (UniqueName: \"kubernetes.io/projected/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-kube-api-access-6hkbn\") pod \"maas-api-revoke-sub-1adf9edd-7w7ph\" (UID: \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"
Oct 30 18:56:26.935090 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.935047    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"]
Oct 30 18:56:26.960351 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.960322    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"
Oct 30 18:56:26.960523 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.960364    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4" event={"ID":"b0d15b24-c4d3-40a2-9203-9db14250fa7c","Type":"ContainerDied","Data":"632e83eb97bc2025012f1265eb81357bd2edeaba1f3a40247a136de512837a5a"}
Oct 30 18:56:26.960523 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.960416    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="632e83eb97bc2025012f1265eb81357bd2edeaba1f3a40247a136de512837a5a"
Oct 30 18:56:26.992788 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.992749    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1adf9edd-7w7ph\" (UID: \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"
Oct 30 18:56:26.992937 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.992862    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1adf9edd-7w7ph\" (UID: \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"
Oct 30 18:56:26.992937 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.992928    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hkbn\" (UniqueName: \"kubernetes.io/projected/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-kube-api-access-6hkbn\") pod \"maas-api-revoke-sub-1adf9edd-7w7ph\" (UID: \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"
Oct 30 18:56:26.993392 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.993367    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1adf9edd-7w7ph\" (UID: \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"
Oct 30 18:56:26.995545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:26.995523    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1adf9edd-7w7ph\" (UID: \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"
Oct 30 18:56:27.049799 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:27.049768    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hkbn\" (UniqueName: \"kubernetes.io/projected/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-kube-api-access-6hkbn\") pod \"maas-api-revoke-sub-1adf9edd-7w7ph\" (UID: \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"
Oct 30 18:56:27.143090 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:27.143061    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"
Oct 30 18:56:27.516699 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:27.516671    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"]
Oct 30 18:56:27.518491 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:56:27.518459    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31dbdbfc_71a6_4f66_bfd9_68b3146b0f19.slice/crio-b85fec9351d232e66bd2d0fa8760e4bbf2ad3333bb6608a4c74604b954dae64a WatchSource:0}: Error finding container b85fec9351d232e66bd2d0fa8760e4bbf2ad3333bb6608a4c74604b954dae64a: Status 404 returned error can't find the container with id b85fec9351d232e66bd2d0fa8760e4bbf2ad3333bb6608a4c74604b954dae64a
Oct 30 18:56:27.969393 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:27.969353    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph" event={"ID":"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19","Type":"ContainerStarted","Data":"ec16f98a3c3bca299d42f69a7de99960fe5e60f13d1e5246ffbea128d8346c7b"}
Oct 30 18:56:27.969599 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:27.969423    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph" event={"ID":"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19","Type":"ContainerStarted","Data":"b85fec9351d232e66bd2d0fa8760e4bbf2ad3333bb6608a4c74604b954dae64a"}
Oct 30 18:56:28.084263 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:28.084197    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph" podStartSLOduration=2.084180173 podStartE2EDuration="2.084180173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:28.081854088 +0000 UTC m=+1026.725021703" watchObservedRunningTime="2026-10-30 18:56:28.084180173 +0000 UTC m=+1026.727347787"
Oct 30 18:56:28.978618 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:28.978580    6014 generic.go:356] "Generic (PLEG): container finished" podID="31dbdbfc-71a6-4f66-bfd9-68b3146b0f19" containerID="ec16f98a3c3bca299d42f69a7de99960fe5e60f13d1e5246ffbea128d8346c7b" exitCode=0
Oct 30 18:56:28.979009 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:28.978655    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph" event={"ID":"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19","Type":"ContainerDied","Data":"ec16f98a3c3bca299d42f69a7de99960fe5e60f13d1e5246ffbea128d8346c7b"}
Oct 30 18:56:30.153060 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:30.153036    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"
Oct 30 18:56:30.231811 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:30.231780    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-xks-ca\") pod \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\" (UID: \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\") "
Oct 30 18:56:30.231985 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:30.231831    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-openshift-ca\") pod \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\" (UID: \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\") "
Oct 30 18:56:30.231985 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:30.231960    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-kube-api-access-6hkbn\" (UniqueName: \"kubernetes.io/projected/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-kube-api-access-6hkbn\") pod \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\" (UID: \"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19\") "
Oct 30 18:56:30.232218 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:30.232194    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-openshift-ca" pod "31dbdbfc-71a6-4f66-bfd9-68b3146b0f19" (UID: "31dbdbfc-71a6-4f66-bfd9-68b3146b0f19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:30.232379 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:30.232365    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:30.234080 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:30.234060    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-xks-ca" pod "31dbdbfc-71a6-4f66-bfd9-68b3146b0f19" (UID: "31dbdbfc-71a6-4f66-bfd9-68b3146b0f19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:30.234170 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:30.234126    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-kube-api-access-6hkbn" pod "31dbdbfc-71a6-4f66-bfd9-68b3146b0f19" (UID: "31dbdbfc-71a6-4f66-bfd9-68b3146b0f19"). InnerVolumeSpecName "kube-api-access-6hkbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:30.333398 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:30.333364    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hkbn\" (UniqueName: \"kubernetes.io/projected/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-kube-api-access-6hkbn\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:30.333398 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:30.333393    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:30.993513 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:30.993482    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"
Oct 30 18:56:30.993706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:30.993482    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph" event={"ID":"31dbdbfc-71a6-4f66-bfd9-68b3146b0f19","Type":"ContainerDied","Data":"b85fec9351d232e66bd2d0fa8760e4bbf2ad3333bb6608a4c74604b954dae64a"}
Oct 30 18:56:30.993706 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:30.993611    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b85fec9351d232e66bd2d0fa8760e4bbf2ad3333bb6608a4c74604b954dae64a"
Oct 30 18:56:38.126478 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.126448    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"]
Oct 30 18:56:38.127059 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.127046    6014 cpu_manager.go:414] "containerMap: removing container" podUID="31dbdbfc-71a6-4f66-bfd9-68b3146b0f19" containerName="revoke-keys"
Oct 30 18:56:38.127110 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.127061    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31dbdbfc-71a6-4f66-bfd9-68b3146b0f19" containerName="revoke-keys"
Oct 30 18:56:38.127191 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.127181    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="31dbdbfc-71a6-4f66-bfd9-68b3146b0f19" containerName="revoke-keys"
Oct 30 18:56:38.130512 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.130496    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"
Oct 30 18:56:38.232986 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.232956    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:56:38.232986 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.232968    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:38.235903 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.235881    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"]
Oct 30 18:56:38.328522 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.328492    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a7f9644-lb98g\" (UID: \"0a6b4585-367a-403f-be63-bd434498bc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"
Oct 30 18:56:38.328714 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.328680    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a7f9644-lb98g\" (UID: \"0a6b4585-367a-403f-be63-bd434498bc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"
Oct 30 18:56:38.328757 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.328727    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9zdx\" (UniqueName: \"kubernetes.io/projected/0a6b4585-367a-403f-be63-bd434498bc54-kube-api-access-b9zdx\") pod \"maas-api-revoke-sub-3a7f9644-lb98g\" (UID: \"0a6b4585-367a-403f-be63-bd434498bc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"
Oct 30 18:56:38.429742 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.429635    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a7f9644-lb98g\" (UID: \"0a6b4585-367a-403f-be63-bd434498bc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"
Oct 30 18:56:38.429928 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.429805    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a7f9644-lb98g\" (UID: \"0a6b4585-367a-403f-be63-bd434498bc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"
Oct 30 18:56:38.429928 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.429848    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9zdx\" (UniqueName: \"kubernetes.io/projected/0a6b4585-367a-403f-be63-bd434498bc54-kube-api-access-b9zdx\") pod \"maas-api-revoke-sub-3a7f9644-lb98g\" (UID: \"0a6b4585-367a-403f-be63-bd434498bc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"
Oct 30 18:56:38.430333 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.430312    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a7f9644-lb98g\" (UID: \"0a6b4585-367a-403f-be63-bd434498bc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"
Oct 30 18:56:38.432307 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.432286    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a7f9644-lb98g\" (UID: \"0a6b4585-367a-403f-be63-bd434498bc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"
Oct 30 18:56:38.487470 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.487438    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9zdx\" (UniqueName: \"kubernetes.io/projected/0a6b4585-367a-403f-be63-bd434498bc54-kube-api-access-b9zdx\") pod \"maas-api-revoke-sub-3a7f9644-lb98g\" (UID: \"0a6b4585-367a-403f-be63-bd434498bc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"
Oct 30 18:56:38.743927 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:38.743840    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"
Oct 30 18:56:39.120025 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:39.119992    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"]
Oct 30 18:56:39.121627 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:56:39.121596    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6b4585_367a_403f_be63_bd434498bc54.slice/crio-8ae1668f25940a03170cc6f4bfc7f2ea6d451946f56c1436828252c66ce3ad98 WatchSource:0}: Error finding container 8ae1668f25940a03170cc6f4bfc7f2ea6d451946f56c1436828252c66ce3ad98: Status 404 returned error can't find the container with id 8ae1668f25940a03170cc6f4bfc7f2ea6d451946f56c1436828252c66ce3ad98
Oct 30 18:56:40.054417 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:40.054384    6014 generic.go:356] "Generic (PLEG): container finished" podID="0a6b4585-367a-403f-be63-bd434498bc54" containerID="02b2137b89c111b507a82ae452a012d123a7d2d95dcdf95c777f40e90c383971" exitCode=0
Oct 30 18:56:40.054806 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:40.054464    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g" event={"ID":"0a6b4585-367a-403f-be63-bd434498bc54","Type":"ContainerDied","Data":"02b2137b89c111b507a82ae452a012d123a7d2d95dcdf95c777f40e90c383971"}
Oct 30 18:56:40.054806 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:40.054515    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g" event={"ID":"0a6b4585-367a-403f-be63-bd434498bc54","Type":"ContainerStarted","Data":"8ae1668f25940a03170cc6f4bfc7f2ea6d451946f56c1436828252c66ce3ad98"}
Oct 30 18:56:41.222487 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:41.222455    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"
Oct 30 18:56:41.259576 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:41.259539    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-xks-ca\") pod \"0a6b4585-367a-403f-be63-bd434498bc54\" (UID: \"0a6b4585-367a-403f-be63-bd434498bc54\") "
Oct 30 18:56:41.259761 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:41.259618    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a6b4585-367a-403f-be63-bd434498bc54-kube-api-access-b9zdx\" (UniqueName: \"kubernetes.io/projected/0a6b4585-367a-403f-be63-bd434498bc54-kube-api-access-b9zdx\") pod \"0a6b4585-367a-403f-be63-bd434498bc54\" (UID: \"0a6b4585-367a-403f-be63-bd434498bc54\") "
Oct 30 18:56:41.259761 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:41.259721    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-openshift-ca\") pod \"0a6b4585-367a-403f-be63-bd434498bc54\" (UID: \"0a6b4585-367a-403f-be63-bd434498bc54\") "
Oct 30 18:56:41.260067 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:41.260045    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-openshift-ca" pod "0a6b4585-367a-403f-be63-bd434498bc54" (UID: "0a6b4585-367a-403f-be63-bd434498bc54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:41.261945 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:41.261921    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-xks-ca" pod "0a6b4585-367a-403f-be63-bd434498bc54" (UID: "0a6b4585-367a-403f-be63-bd434498bc54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:41.262034 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:41.261951    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6b4585-367a-403f-be63-bd434498bc54-kube-api-access-b9zdx" pod "0a6b4585-367a-403f-be63-bd434498bc54" (UID: "0a6b4585-367a-403f-be63-bd434498bc54"). InnerVolumeSpecName "kube-api-access-b9zdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:41.360979 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:41.360942    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:41.360979 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:41.360973    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9zdx\" (UniqueName: \"kubernetes.io/projected/0a6b4585-367a-403f-be63-bd434498bc54-kube-api-access-b9zdx\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:41.360979 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:41.360984    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6b4585-367a-403f-be63-bd434498bc54-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:42.068275 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:42.068241    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g" event={"ID":"0a6b4585-367a-403f-be63-bd434498bc54","Type":"ContainerDied","Data":"8ae1668f25940a03170cc6f4bfc7f2ea6d451946f56c1436828252c66ce3ad98"}
Oct 30 18:56:42.068275 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:42.068265    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"
Oct 30 18:56:42.068492 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:56:42.068290    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ae1668f25940a03170cc6f4bfc7f2ea6d451946f56c1436828252c66ce3ad98"
Oct 30 18:56:53.258500 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:56:53.258461    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:57:00.521134 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:00.521099    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:57:01.865084 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.865044    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"]
Oct 30 18:57:01.865711 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.865692    6014 cpu_manager.go:414] "containerMap: removing container" podUID="0a6b4585-367a-403f-be63-bd434498bc54" containerName="revoke-keys"
Oct 30 18:57:01.865711 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.865711    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a6b4585-367a-403f-be63-bd434498bc54" containerName="revoke-keys"
Oct 30 18:57:01.865854 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.865837    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a6b4585-367a-403f-be63-bd434498bc54" containerName="revoke-keys"
Oct 30 18:57:01.871069 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.871045    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:01.934517 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.934484    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf524\" (UniqueName: \"kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-kube-api-access-rf524\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:01.934517 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.934524    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:01.934770 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.934586    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:01.934770 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.934630    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:01.934770 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.934700    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:01.934770 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.934727    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:01.935008 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.934860    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:01.935008 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.934904    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:01.935115 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:01.935010    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.036475 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.036440    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf524\" (UniqueName: \"kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-kube-api-access-rf524\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.036475 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.036478    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.036754 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.036499    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.036754 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.036523    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.036754 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.036544    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.036754 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.036558    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.036754 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.036642    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.036754 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.036690    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.037084 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.036767    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.037084 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.036880    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.037084 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.036925    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.037084 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.036976    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.037370 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.037350    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.037616 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.037592    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.039152 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.039119    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.039534 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.039514    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.071500 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.071458    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"]
Oct 30 18:57:02.094547 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.094515    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.094741 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.094633    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf524\" (UniqueName: \"kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-kube-api-access-rf524\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.192880 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.192795    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:02.622879 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.622834    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"]
Oct 30 18:57:02.623861 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:57:02.623822    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f71e791_7dc8_41b8_aa8c_e3c4b21779cb.slice/crio-6a041edf64d41292e4bc0138dd989ee8d088cffa72b9d5ad56af113e3a604560 WatchSource:0}: Error finding container 6a041edf64d41292e4bc0138dd989ee8d088cffa72b9d5ad56af113e3a604560: Status 404 returned error can't find the container with id 6a041edf64d41292e4bc0138dd989ee8d088cffa72b9d5ad56af113e3a604560
Oct 30 18:57:02.626391 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.626356    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:57:02.626485 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.626423    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:57:02.626485 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.626452    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:57:02.948957 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.948857    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"]
Oct 30 18:57:02.954075 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.954052    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:02.975903 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.975873    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"]
Oct 30 18:57:02.980048 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:02.980026    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.011840 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.011812    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"]
Oct 30 18:57:03.050128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.049528    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.050128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.049590    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpbxv\" (UniqueName: \"kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-kube-api-access-xpbxv\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.050128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.049617    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.050128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.049673    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.050128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.049703    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43fd9a91-35f5-454e-95ad-8f2c446ed639-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.050128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.049750    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.050128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.049778    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.050128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.049848    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.050128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.049871    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.050128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.049920    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f961678-957e-4e77-94c2-c551cb6627cb-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.050128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.050043    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.050128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.050077    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.050128 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.050131    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdpgr\" (UniqueName: \"kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-kube-api-access-jdpgr\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.051180 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.050198    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f961678-957e-4e77-94c2-c551cb6627cb-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.051180 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.050274    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.051180 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.050314    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.051180 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.050366    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.051180 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.050417    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.062054 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.061031    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"]
Oct 30 18:57:03.172164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.171928    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.172164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172002    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.172164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172053    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f961678-957e-4e77-94c2-c551cb6627cb-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.172164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172102    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.172164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172131    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.172164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172176    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdpgr\" (UniqueName: \"kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-kube-api-access-jdpgr\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.172647 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172275    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f961678-957e-4e77-94c2-c551cb6627cb-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.172647 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172369    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.172647 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172410    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.172647 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172458    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.172647 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172524    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.172647 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172586    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.172647 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172628    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpbxv\" (UniqueName: \"kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-kube-api-access-xpbxv\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.173011 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172655    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.173011 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172727    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.173011 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172757    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43fd9a91-35f5-454e-95ad-8f2c446ed639-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.173011 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172787    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.173011 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172815    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.173011 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.172845    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.173524 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.173500    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.173596 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.173536    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f961678-957e-4e77-94c2-c551cb6627cb-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.173952 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.173926    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.174103 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.174082    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.174674 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.174646    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43fd9a91-35f5-454e-95ad-8f2c446ed639-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.175078 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.175054    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.175391 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.175367    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.175522 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.175500    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f961678-957e-4e77-94c2-c551cb6627cb-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.175960 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.175936    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.176074 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.176007    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.176336 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.176311    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.176792 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.176764    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.176857 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.176800    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.229430 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.229352    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdpgr\" (UniqueName: \"kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-kube-api-access-jdpgr\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.229859 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.229832    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.230273 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.230243    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.230490 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.230473    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpbxv\" (UniqueName: \"kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-kube-api-access-xpbxv\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.232198 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.232170    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g" event={"ID":"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb","Type":"ContainerStarted","Data":"74e9bcfbbf4cae0c326c2ba13d0b958bade252b6a100722f793c0fd20c22d9e3"}
Oct 30 18:57:03.232301 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.232258    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g" event={"ID":"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb","Type":"ContainerStarted","Data":"6a041edf64d41292e4bc0138dd989ee8d088cffa72b9d5ad56af113e3a604560"}
Oct 30 18:57:03.272466 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.272427    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:03.294954 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.294916    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:03.381251 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.380636    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g" podStartSLOduration=2.380614115 podStartE2EDuration="2.380614115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:03.361612846 +0000 UTC m=+1062.004780461" watchObservedRunningTime="2026-10-30 18:57:03.380614115 +0000 UTC m=+1062.023781731"
Oct 30 18:57:03.706753 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.706729    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"]
Oct 30 18:57:03.708057 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:57:03.708033    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f961678_957e_4e77_94c2_c551cb6627cb.slice/crio-a997e404756235f4054ba84110651822e0cdf5900c450f4104694600bb5308fa WatchSource:0}: Error finding container a997e404756235f4054ba84110651822e0cdf5900c450f4104694600bb5308fa: Status 404 returned error can't find the container with id a997e404756235f4054ba84110651822e0cdf5900c450f4104694600bb5308fa
Oct 30 18:57:03.710575 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.710546    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:57:03.710646 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.710610    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:57:03.710646 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.710639    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:57:03.775665 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.775502    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"]
Oct 30 18:57:03.792574 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:57:03.792499    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fd9a91_35f5_454e_95ad_8f2c446ed639.slice/crio-1c6fb4d7f6e72d8d508dc43c6436a1d27f3dff2d6ffcd1403605f6c74222121e WatchSource:0}: Error finding container 1c6fb4d7f6e72d8d508dc43c6436a1d27f3dff2d6ffcd1403605f6c74222121e: Status 404 returned error can't find the container with id 1c6fb4d7f6e72d8d508dc43c6436a1d27f3dff2d6ffcd1403605f6c74222121e
Oct 30 18:57:03.795195 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.795156    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:57:03.795273 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.795245    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:57:03.795321 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:03.795292    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:57:04.194260 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.194218    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:04.195669 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.195638    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:57:04.195812 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.195687    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g" podUID="6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 18:57:04.242241 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.242202    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" event={"ID":"9f961678-957e-4e77-94c2-c551cb6627cb","Type":"ContainerStarted","Data":"9c59324f3efe53422e15b8633234278cb257be52251de28b0f57013d027564d1"}
Oct 30 18:57:04.242454 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.242272    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" event={"ID":"9f961678-957e-4e77-94c2-c551cb6627cb","Type":"ContainerStarted","Data":"a997e404756235f4054ba84110651822e0cdf5900c450f4104694600bb5308fa"}
Oct 30 18:57:04.244249 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.244216    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" event={"ID":"43fd9a91-35f5-454e-95ad-8f2c446ed639","Type":"ContainerStarted","Data":"9c764953ac23ce7d8eda0fb182c49f72043b838338cf62b3f29526310a7fc221"}
Oct 30 18:57:04.244430 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.244273    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" event={"ID":"43fd9a91-35f5-454e-95ad-8f2c446ed639","Type":"ContainerStarted","Data":"1c6fb4d7f6e72d8d508dc43c6436a1d27f3dff2d6ffcd1403605f6c74222121e"}
Oct 30 18:57:04.277844 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.277813    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:04.278269 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.278244    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 18:57:04.278456 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.278433    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" podUID="9f961678-957e-4e77-94c2-c551cb6627cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 18:57:04.295643 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.295608    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:04.297576 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.297549    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 18:57:04.297716 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.297606    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" podUID="43fd9a91-35f5-454e-95ad-8f2c446ed639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 18:57:04.487226 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.484754    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" podStartSLOduration=2.484733397 podStartE2EDuration="2.484733397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:04.483655311 +0000 UTC m=+1063.126822927" watchObservedRunningTime="2026-10-30 18:57:04.484733397 +0000 UTC m=+1063.127901011"
Oct 30 18:57:04.487226 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:04.485899    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" podStartSLOduration=2.4858799 podStartE2EDuration="2.4858799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:04.365014226 +0000 UTC m=+1063.008181832" watchObservedRunningTime="2026-10-30 18:57:04.4858799 +0000 UTC m=+1063.129047517"
Oct 30 18:57:05.208165 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:05.195649    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:57:05.208165 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:05.195731    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g" podUID="6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 18:57:05.276164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:05.275788    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 18:57:05.276164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:05.275850    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" podUID="9f961678-957e-4e77-94c2-c551cb6627cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 18:57:05.311164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:05.300517    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 18:57:05.311164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:05.300574    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" podUID="43fd9a91-35f5-454e-95ad-8f2c446ed639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 18:57:06.196166 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:06.195653    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:57:06.196166 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:06.195728    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g" podUID="6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 18:57:06.274167 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:06.273561    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 18:57:06.274167 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:06.273628    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" podUID="9f961678-957e-4e77-94c2-c551cb6627cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 18:57:06.296177 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:06.295515    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 18:57:06.296177 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:06.295586    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" podUID="43fd9a91-35f5-454e-95ad-8f2c446ed639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 18:57:07.194627 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:07.194581    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:57:07.194920 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:07.194888    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g" podUID="6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 18:57:07.286022 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:07.285980    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 18:57:07.287664 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:07.287626    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" podUID="9f961678-957e-4e77-94c2-c551cb6627cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 18:57:07.299702 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:07.299663    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 18:57:07.299977 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:07.299947    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" podUID="43fd9a91-35f5-454e-95ad-8f2c446ed639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 18:57:08.197164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:08.196609    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:57:08.197164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:08.196677    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g" podUID="6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 18:57:08.275674 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:08.275634    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 18:57:08.275964 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:08.275935    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" podUID="9f961678-957e-4e77-94c2-c551cb6627cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 18:57:08.295636 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:08.295595    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 18:57:08.296261 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:08.296227    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" podUID="43fd9a91-35f5-454e-95ad-8f2c446ed639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 18:57:09.196096 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:09.195606    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:57:09.196096 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:09.195706    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g" podUID="6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 18:57:09.274704 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:09.274664    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 18:57:09.275033 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:09.274987    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" podUID="9f961678-957e-4e77-94c2-c551cb6627cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 18:57:09.303049 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:09.303009    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 18:57:09.303792 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:09.303673    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" podUID="43fd9a91-35f5-454e-95ad-8f2c446ed639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 18:57:10.198567 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:10.198526    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:10.198968 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:10.198944    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:10.199796 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:10.199772    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 18:57:10.273323 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:10.273280    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 18:57:10.273500 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:10.273384    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" podUID="9f961678-957e-4e77-94c2-c551cb6627cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 18:57:10.295893 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:10.295851    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 18:57:10.296016 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:10.295957    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" podUID="43fd9a91-35f5-454e-95ad-8f2c446ed639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 18:57:11.278080 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:11.278049    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:11.278519 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:11.278500    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:11.279386 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:11.279367    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 18:57:11.300053 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:11.300025    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:11.300433 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:11.300403    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:11.301072 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:11.301058    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 18:57:23.258559 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:57:23.258521    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:57:23.885698 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:23.885664    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:57:48.613198 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:48.613107    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:57:52.563830 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.563797    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"]
Oct 30 18:57:52.568314 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.568292    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"
Oct 30 18:57:52.623603 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.623568    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-364e71a8-9vbsj\" (UID: \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"
Oct 30 18:57:52.623850 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.623620    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f57mz\" (UniqueName: \"kubernetes.io/projected/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-kube-api-access-f57mz\") pod \"maas-api-revoke-sub-364e71a8-9vbsj\" (UID: \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"
Oct 30 18:57:52.623850 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.623751    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-364e71a8-9vbsj\" (UID: \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"
Oct 30 18:57:52.669768 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.669737    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:52.669964 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.669882    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:57:52.673118 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.673097    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"]
Oct 30 18:57:52.725048 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.725012    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-364e71a8-9vbsj\" (UID: \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"
Oct 30 18:57:52.725048 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.725056    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f57mz\" (UniqueName: \"kubernetes.io/projected/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-kube-api-access-f57mz\") pod \"maas-api-revoke-sub-364e71a8-9vbsj\" (UID: \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"
Oct 30 18:57:52.725303 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.725110    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-364e71a8-9vbsj\" (UID: \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"
Oct 30 18:57:52.725884 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.725861    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-364e71a8-9vbsj\" (UID: \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"
Oct 30 18:57:52.727662 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.727641    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-364e71a8-9vbsj\" (UID: \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"
Oct 30 18:57:52.782156 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.782099    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f57mz\" (UniqueName: \"kubernetes.io/projected/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-kube-api-access-f57mz\") pod \"maas-api-revoke-sub-364e71a8-9vbsj\" (UID: \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"
Oct 30 18:57:52.887904 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:52.887815    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"
Oct 30 18:57:53.259809 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:57:53.259712    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:57:53.262087 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:53.262065    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"]
Oct 30 18:57:53.263762 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:57:53.263724    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5b6fbb_a1b8_43e6_b847_dd6102e1b603.slice/crio-4f13910fd8cfbe9c5706a5098ed3c065984c9559851732f09ec8ee4a15c3f9a2 WatchSource:0}: Error finding container 4f13910fd8cfbe9c5706a5098ed3c065984c9559851732f09ec8ee4a15c3f9a2: Status 404 returned error can't find the container with id 4f13910fd8cfbe9c5706a5098ed3c065984c9559851732f09ec8ee4a15c3f9a2
Oct 30 18:57:53.706107 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:53.706070    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj" event={"ID":"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603","Type":"ContainerStarted","Data":"724e31b3e39d2b53c80d58ac7ddaca8cedbb0dbc3133b40593e7cd16e147d563"}
Oct 30 18:57:53.706513 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:53.706127    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj" event={"ID":"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603","Type":"ContainerStarted","Data":"4f13910fd8cfbe9c5706a5098ed3c065984c9559851732f09ec8ee4a15c3f9a2"}
Oct 30 18:57:53.819849 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:53.819793    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj" podStartSLOduration=1.819777845 podStartE2EDuration="1.819777845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:53.818848102 +0000 UTC m=+1112.462015716" watchObservedRunningTime="2026-10-30 18:57:53.819777845 +0000 UTC m=+1112.462945458"
Oct 30 18:57:54.713865 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:54.713826    6014 generic.go:356] "Generic (PLEG): container finished" podID="fd5b6fbb-a1b8-43e6-b847-dd6102e1b603" containerID="724e31b3e39d2b53c80d58ac7ddaca8cedbb0dbc3133b40593e7cd16e147d563" exitCode=0
Oct 30 18:57:54.714316 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:54.713918    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj" event={"ID":"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603","Type":"ContainerDied","Data":"724e31b3e39d2b53c80d58ac7ddaca8cedbb0dbc3133b40593e7cd16e147d563"}
Oct 30 18:57:55.871170 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:55.871114    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"
Oct 30 18:57:55.965674 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:55.965631    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-xks-ca\") pod \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\" (UID: \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\") "
Oct 30 18:57:55.965877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:55.965680    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-openshift-ca\") pod \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\" (UID: \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\") "
Oct 30 18:57:55.965877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:55.965716    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-kube-api-access-f57mz\" (UniqueName: \"kubernetes.io/projected/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-kube-api-access-f57mz\") pod \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\" (UID: \"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603\") "
Oct 30 18:57:55.966164 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:55.966115    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-openshift-ca" pod "fd5b6fbb-a1b8-43e6-b847-dd6102e1b603" (UID: "fd5b6fbb-a1b8-43e6-b847-dd6102e1b603"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:55.966396 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:55.966378    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:57:55.967987 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:55.967963    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-xks-ca" pod "fd5b6fbb-a1b8-43e6-b847-dd6102e1b603" (UID: "fd5b6fbb-a1b8-43e6-b847-dd6102e1b603"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:55.967987 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:55.967983    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-kube-api-access-f57mz" pod "fd5b6fbb-a1b8-43e6-b847-dd6102e1b603" (UID: "fd5b6fbb-a1b8-43e6-b847-dd6102e1b603"). InnerVolumeSpecName "kube-api-access-f57mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:56.067566 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:56.067528    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:57:56.067755 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:56.067583    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f57mz\" (UniqueName: \"kubernetes.io/projected/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603-kube-api-access-f57mz\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:57:56.728686 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:56.728655    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"
Oct 30 18:57:56.728686 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:56.728672    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj" event={"ID":"fd5b6fbb-a1b8-43e6-b847-dd6102e1b603","Type":"ContainerDied","Data":"4f13910fd8cfbe9c5706a5098ed3c065984c9559851732f09ec8ee4a15c3f9a2"}
Oct 30 18:57:56.728900 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:56.728719    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f13910fd8cfbe9c5706a5098ed3c065984c9559851732f09ec8ee4a15c3f9a2"
Oct 30 18:57:57.279778 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:57.279734    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"]
Oct 30 18:57:57.280300 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:57:57.279997    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerName="istio-proxy" containerID="cri-o://222fb88bed456a87e5fb009c9decd9e97f4cb913b6f4e1d56c80af97e28fc66f" gracePeriod=30
Oct 30 18:58:00.650023 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:00.649979    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jzvkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:58:00.650450 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:00.650056    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:58:02.539441 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.539411    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:58:02.647370 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.647333    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-socket\") pod \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") "
Oct 30 18:58:02.647554 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.647402    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-envoy\") pod \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") "
Oct 30 18:58:02.647554 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.647462    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istiod-ca-cert\") pod \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") "
Oct 30 18:58:02.647643 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.647568    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-token\" (UniqueName: \"kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-token\") pod \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") "
Oct 30 18:58:02.647643 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.647606    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-data\") pod \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") "
Oct 30 18:58:02.647734 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.647653    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-socket" pod "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" (UID: "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:02.647788 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.647737    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-certs\") pod \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") "
Oct 30 18:58:02.647788 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.647765    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-podinfo\") pod \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") "
Oct 30 18:58:02.647906 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.647813    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-credential-socket\") pod \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") "
Oct 30 18:58:02.647906 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.647865    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-kube-api-access-7cjdw\" (UniqueName: \"kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-kube-api-access-7cjdw\") pod \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\" (UID: \"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a\") "
Oct 30 18:58:02.648006 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.647918    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-data" pod "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" (UID: "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:02.648006 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.647982    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istiod-ca-cert" pod "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" (UID: "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:02.648877 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.648472    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-certs" pod "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" (UID: "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:02.649055 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.649034    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-credential-socket" pod "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" (UID: "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:02.650458 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.650429    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-podinfo" pod "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" (UID: "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:58:02.650586 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.650456    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-token" pod "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" (UID: "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:02.651258 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.651232    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-kube-api-access-7cjdw" pod "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" (UID: "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a"). InnerVolumeSpecName "kube-api-access-7cjdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:02.651374 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.651256    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-envoy" pod "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" (UID: "6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:02.658669 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.658638    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:58:02.658669 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.658671    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:58:02.658847 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.658686    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:58:02.760299 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.760257    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:58:02.760299 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.760302    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:58:02.760545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.760319    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:58:02.760545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.760333    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:58:02.760545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.760347    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cjdw\" (UniqueName: \"kubernetes.io/projected/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-kube-api-access-7cjdw\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:58:02.760545 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.760362    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:58:02.770622 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.770589    6014 generic.go:356] "Generic (PLEG): container finished" podID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerID="222fb88bed456a87e5fb009c9decd9e97f4cb913b6f4e1d56c80af97e28fc66f" exitCode=0
Oct 30 18:58:02.770818 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.770671    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"
Oct 30 18:58:02.770818 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.770687    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" event={"ID":"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a","Type":"ContainerDied","Data":"222fb88bed456a87e5fb009c9decd9e97f4cb913b6f4e1d56c80af97e28fc66f"}
Oct 30 18:58:02.770818 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.770769    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj" event={"ID":"6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a","Type":"ContainerDied","Data":"17280558be478dcb22358716d8ca57c450397b5a605bab1964dd37767df3b6f1"}
Oct 30 18:58:02.770818 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.770799    6014 scope.go:122] "RemoveContainer" containerID="222fb88bed456a87e5fb009c9decd9e97f4cb913b6f4e1d56c80af97e28fc66f"
Oct 30 18:58:02.783408 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.783389    6014 scope.go:122] "RemoveContainer" containerID="37dd2c746f28d42e081fa5dec673eeebc456a3f6c309995aac6073e281e920c3"
Oct 30 18:58:02.797362 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.797246    6014 scope.go:122] "RemoveContainer" containerID="222fb88bed456a87e5fb009c9decd9e97f4cb913b6f4e1d56c80af97e28fc66f"
Oct 30 18:58:02.798098 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:58:02.797932    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222fb88bed456a87e5fb009c9decd9e97f4cb913b6f4e1d56c80af97e28fc66f\": container with ID starting with 222fb88bed456a87e5fb009c9decd9e97f4cb913b6f4e1d56c80af97e28fc66f not found: ID does not exist" containerID="222fb88bed456a87e5fb009c9decd9e97f4cb913b6f4e1d56c80af97e28fc66f"
Oct 30 18:58:02.798098 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.797970    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222fb88bed456a87e5fb009c9decd9e97f4cb913b6f4e1d56c80af97e28fc66f"} err="failed to get container status \"222fb88bed456a87e5fb009c9decd9e97f4cb913b6f4e1d56c80af97e28fc66f\": rpc error: code = NotFound desc = could not find container \"222fb88bed456a87e5fb009c9decd9e97f4cb913b6f4e1d56c80af97e28fc66f\": container with ID starting with 222fb88bed456a87e5fb009c9decd9e97f4cb913b6f4e1d56c80af97e28fc66f not found: ID does not exist"
Oct 30 18:58:02.798098 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.797994    6014 scope.go:122] "RemoveContainer" containerID="37dd2c746f28d42e081fa5dec673eeebc456a3f6c309995aac6073e281e920c3"
Oct 30 18:58:02.798863 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:58:02.798838    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37dd2c746f28d42e081fa5dec673eeebc456a3f6c309995aac6073e281e920c3\": container with ID starting with 37dd2c746f28d42e081fa5dec673eeebc456a3f6c309995aac6073e281e920c3 not found: ID does not exist" containerID="37dd2c746f28d42e081fa5dec673eeebc456a3f6c309995aac6073e281e920c3"
Oct 30 18:58:02.798951 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.798872    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37dd2c746f28d42e081fa5dec673eeebc456a3f6c309995aac6073e281e920c3"} err="failed to get container status \"37dd2c746f28d42e081fa5dec673eeebc456a3f6c309995aac6073e281e920c3\": rpc error: code = NotFound desc = could not find container \"37dd2c746f28d42e081fa5dec673eeebc456a3f6c309995aac6073e281e920c3\": container with ID starting with 37dd2c746f28d42e081fa5dec673eeebc456a3f6c309995aac6073e281e920c3 not found: ID does not exist"
Oct 30 18:58:02.935783 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.935687    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"]
Oct 30 18:58:02.940749 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:02.940716    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jzvkj"]
Oct 30 18:58:03.476677 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:03.476646    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" path="/var/lib/kubelet/pods/6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a/volumes"
Oct 30 18:58:23.258768 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:58:23.258734    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:58:25.369427 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.369397    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"]
Oct 30 18:58:25.369988 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.369974    6014 cpu_manager.go:414] "containerMap: removing container" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerName="istio-proxy"
Oct 30 18:58:25.370031 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.369989    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerName="istio-proxy"
Oct 30 18:58:25.370397 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.370373    6014 cpu_manager.go:414] "containerMap: removing container" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerName="istio-proxy"
Oct 30 18:58:25.370472 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.370398    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerName="istio-proxy"
Oct 30 18:58:25.370528 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.370470    6014 cpu_manager.go:414] "containerMap: removing container" podUID="fd5b6fbb-a1b8-43e6-b847-dd6102e1b603" containerName="revoke-keys"
Oct 30 18:58:25.370528 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.370481    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5b6fbb-a1b8-43e6-b847-dd6102e1b603" containerName="revoke-keys"
Oct 30 18:58:25.382121 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.382090    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerName="istio-proxy"
Oct 30 18:58:25.382316 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.382133    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5b6fbb-a1b8-43e6-b847-dd6102e1b603" containerName="revoke-keys"
Oct 30 18:58:25.387438 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.387411    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.432505 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.432467    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"]
Oct 30 18:58:25.477462 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.477433    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f5763a\"/\"distinct-2-w0-f5763a-kserve-self-signed-certs\""
Oct 30 18:58:25.490094 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.490058    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-dshm\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.490267 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.490111    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-model-cache\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.490309 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.490272    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tmp-dir\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.490350 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.490311    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnxn\" (UniqueName: \"kubernetes.io/projected/1fb951bd-2257-46f6-8adc-50f8fa5f7082-kube-api-access-ghnxn\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.490438 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.490425    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-home\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.490473 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.490458    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tls-certs\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.592177 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.592123    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-home\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.592365 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.592191    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tls-certs\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.592365 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.592250    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-dshm\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.592365 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.592291    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-model-cache\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.592504 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.592381    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tmp-dir\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.592504 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.592425    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnxn\" (UniqueName: \"kubernetes.io/projected/1fb951bd-2257-46f6-8adc-50f8fa5f7082-kube-api-access-ghnxn\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.592595 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.592545    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-home\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.592666 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.592636    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-model-cache\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.592724 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.592689    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tmp-dir\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.594757 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.594732    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-dshm\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.594882 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.594865    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tls-certs\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.650607 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.650530    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnxn\" (UniqueName: \"kubernetes.io/projected/1fb951bd-2257-46f6-8adc-50f8fa5f7082-kube-api-access-ghnxn\") pod \"distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:25.709979 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:25.709944    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:26.137054 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:26.137027    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"]
Oct 30 18:58:26.138453 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:58:26.138417    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fb951bd_2257_46f6_8adc_50f8fa5f7082.slice/crio-4840062f3a61cf7b39280f7cb8e8eea4566d570eaffa1d3c7c72810ef5b52449 WatchSource:0}: Error finding container 4840062f3a61cf7b39280f7cb8e8eea4566d570eaffa1d3c7c72810ef5b52449: Status 404 returned error can't find the container with id 4840062f3a61cf7b39280f7cb8e8eea4566d570eaffa1d3c7c72810ef5b52449
Oct 30 18:58:26.971957 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:26.971920    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" event={"ID":"1fb951bd-2257-46f6-8adc-50f8fa5f7082","Type":"ContainerStarted","Data":"7a178c73a4f4ce7945453f7fb76a903643fc6a8a06cb04d428072aa2c0e1e175"}
Oct 30 18:58:26.972384 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:26.971971    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" event={"ID":"1fb951bd-2257-46f6-8adc-50f8fa5f7082","Type":"ContainerStarted","Data":"4840062f3a61cf7b39280f7cb8e8eea4566d570eaffa1d3c7c72810ef5b52449"}
Oct 30 18:58:27.088845 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:27.088787    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podStartSLOduration=2.088771717 podStartE2EDuration="2.088771717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:27.085796665 +0000 UTC m=+1145.728964279" watchObservedRunningTime="2026-10-30 18:58:27.088771717 +0000 UTC m=+1145.731939324"
Oct 30 18:58:27.711047 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:27.711006    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:32.657490 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.657447    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:58:32.859874 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.859835    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"]
Oct 30 18:58:32.860633 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.860615    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cdf1a74-fe4f-4e2e-bc7e-0f7d259e4a7a" containerName="istio-proxy"
Oct 30 18:58:32.864249 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.864225    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:32.959795 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.959713    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"]
Oct 30 18:58:32.992716 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.992676    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:32.993013 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.992991    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58115155-061f-40f8-b45d-ac30c33b5f2c-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:32.993354 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.993321    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:32.993495 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.993389    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8tv\" (UniqueName: \"kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-kube-api-access-vf8tv\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:32.993495 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.993435    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:32.993495 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.993486    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:32.993675 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.993516    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:32.993675 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.993580    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:32.993799 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:32.993768    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.095356 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.095318    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.095553 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.095384    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58115155-061f-40f8-b45d-ac30c33b5f2c-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.095553 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.095445    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.095553 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.095496    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8tv\" (UniqueName: \"kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-kube-api-access-vf8tv\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.095553 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.095539    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.095784 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.095585    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.095784 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.095615    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.095784 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.095681    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.095784 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.095720    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.095784 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.095764    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.096042 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.095960    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.096042 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.096005    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.096194 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.096129    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.096430 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.096411    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58115155-061f-40f8-b45d-ac30c33b5f2c-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.098088 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.098063    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.098287 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.098260    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.153703 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.153662    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.153854 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.153783    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8tv\" (UniqueName: \"kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-kube-api-access-vf8tv\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.179376 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.179345    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:33.622452 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.622423    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"]
Oct 30 18:58:33.624559 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:58:33.624529    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58115155_061f_40f8_b45d_ac30c33b5f2c.slice/crio-63d1a3b797e3e7ec4c1e6dba0d2103a6d2c5a4e72ae3d64b1e8375987379754b WatchSource:0}: Error finding container 63d1a3b797e3e7ec4c1e6dba0d2103a6d2c5a4e72ae3d64b1e8375987379754b: Status 404 returned error can't find the container with id 63d1a3b797e3e7ec4c1e6dba0d2103a6d2c5a4e72ae3d64b1e8375987379754b
Oct 30 18:58:33.627267 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.627230    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:33.627339 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.627303    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:33.627339 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:33.627333    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:34.024590 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:34.024498    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" event={"ID":"58115155-061f-40f8-b45d-ac30c33b5f2c","Type":"ContainerStarted","Data":"0dfc5a9bcef0d59949e64cf3f16172b6889ea1f1531fd61b41b044c2e6fedcd8"}
Oct 30 18:58:34.025174 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:34.025100    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" event={"ID":"58115155-061f-40f8-b45d-ac30c33b5f2c","Type":"ContainerStarted","Data":"63d1a3b797e3e7ec4c1e6dba0d2103a6d2c5a4e72ae3d64b1e8375987379754b"}
Oct 30 18:58:34.146200 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:34.146131    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" podStartSLOduration=2.146115685 podStartE2EDuration="2.146115685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:34.14342483 +0000 UTC m=+1152.786592447" watchObservedRunningTime="2026-10-30 18:58:34.146115685 +0000 UTC m=+1152.789283335"
Oct 30 18:58:34.180511 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:34.180474    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:34.182981 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:34.182946    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:58:34.183271 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:34.183241    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" podUID="58115155-061f-40f8-b45d-ac30c33b5f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 18:58:35.179958 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:35.179913    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:58:35.180410 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:35.179979    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" podUID="58115155-061f-40f8-b45d-ac30c33b5f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 18:58:35.710754 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:35.710716    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:35.724101 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:35.724073    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:36.078249 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:36.078029    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 18:58:36.180547 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:36.180506    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:58:36.180965 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:36.180583    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" podUID="58115155-061f-40f8-b45d-ac30c33b5f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 18:58:37.187151 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:37.184252    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:58:37.187151 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:37.184323    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" podUID="58115155-061f-40f8-b45d-ac30c33b5f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 18:58:38.180717 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:38.180672    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:58:38.180901 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:38.180731    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" podUID="58115155-061f-40f8-b45d-ac30c33b5f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 18:58:39.180692 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:39.180656    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:58:39.181095 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:39.180717    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" podUID="58115155-061f-40f8-b45d-ac30c33b5f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 18:58:40.180396 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:40.180358    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:58:40.180572 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:40.180421    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" podUID="58115155-061f-40f8-b45d-ac30c33b5f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 18:58:41.185515 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:41.185480    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:41.185965 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:41.185893    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:41.186551 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:41.186519    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 18:58:48.316273 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.316237    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"]
Oct 30 18:58:48.320559 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.320539    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.426345 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.426306    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"]
Oct 30 18:58:48.433920 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.433889    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.434178 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.433943    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.434178 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.434157    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.434313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.434253    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.434313 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.434287    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.434425 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.434330    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.434425 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.434392    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461846ef-bc00-4cea-93f6-5d8deb63e544-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.434425 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.434414    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnhch\" (UniqueName: \"kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-kube-api-access-cnhch\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.434649 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.434444    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.535752 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.535715    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.535945 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.535756    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.535945 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.535796    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.535945 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.535846    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.536105 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.535947    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461846ef-bc00-4cea-93f6-5d8deb63e544-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.536269 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.536246    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.536269 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.536266    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.536455 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.535972    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnhch\" (UniqueName: \"kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-kube-api-access-cnhch\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.536501 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.536460    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.536543 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.536522    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.536584 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.536559    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.536740 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.536709    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461846ef-bc00-4cea-93f6-5d8deb63e544-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.536880 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.536765    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.536880 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.536806    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.538386 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.538363    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.539004 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.538983    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.593001 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.592928    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.593133 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.593023    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnhch\" (UniqueName: \"kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-kube-api-access-cnhch\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:48.635437 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:48.635407    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:49.068903 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:49.068878    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"]
Oct 30 18:58:49.071191 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:58:49.071126    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461846ef_bc00_4cea_93f6_5d8deb63e544.slice/crio-820d5a87087acffbe12dd32c5617feb6c327886de56d59ee16e7a80cb112caa5 WatchSource:0}: Error finding container 820d5a87087acffbe12dd32c5617feb6c327886de56d59ee16e7a80cb112caa5: Status 404 returned error can't find the container with id 820d5a87087acffbe12dd32c5617feb6c327886de56d59ee16e7a80cb112caa5
Oct 30 18:58:49.073639 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:49.073607    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:49.073694 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:49.073674    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:49.073732 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:49.073704    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:49.148390 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:49.148359    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7" event={"ID":"461846ef-bc00-4cea-93f6-5d8deb63e544","Type":"ContainerStarted","Data":"820d5a87087acffbe12dd32c5617feb6c327886de56d59ee16e7a80cb112caa5"}
Oct 30 18:58:50.180488 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:50.180451    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7" event={"ID":"461846ef-bc00-4cea-93f6-5d8deb63e544","Type":"ContainerStarted","Data":"b4777602ec192006168868ab0eb5eb49b3379fa82607e36ed889b41118113f52"}
Oct 30 18:58:50.372168 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:50.370171    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7" podStartSLOduration=2.370132836 podStartE2EDuration="2.370132836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:50.369683388 +0000 UTC m=+1169.012851015" watchObservedRunningTime="2026-10-30 18:58:50.370132836 +0000 UTC m=+1169.013300452"
Oct 30 18:58:50.636273 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:50.636237    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:50.637881 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:50.637852    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:58:50.638038 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:50.637920    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7" podUID="461846ef-bc00-4cea-93f6-5d8deb63e544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 18:58:51.636782 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:51.636745    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:58:51.637220 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:51.636811    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7" podUID="461846ef-bc00-4cea-93f6-5d8deb63e544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 18:58:52.636620 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:52.636581    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:58:52.636807 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:52.636663    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7" podUID="461846ef-bc00-4cea-93f6-5d8deb63e544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 18:58:53.280851 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:58:53.280806    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:58:53.636527 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:53.636487    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:58:53.636713 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:53.636543    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7" podUID="461846ef-bc00-4cea-93f6-5d8deb63e544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 18:58:54.636250 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:54.636208    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:58:54.636648 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:54.636271    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7" podUID="461846ef-bc00-4cea-93f6-5d8deb63e544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 18:58:55.636083 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:55.636034    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:58:55.636351 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:55.636093    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7" podUID="461846ef-bc00-4cea-93f6-5d8deb63e544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 18:58:56.865683 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:56.865648    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:56.866225 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:56.866030    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:58:56.867548 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:58:56.867526    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 18:59:01.953237 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:01.953158    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"]
Oct 30 18:59:01.956882 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:01.956860    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.011671 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.011626    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-66bdc0\"/\"openshift-service-ca.crt\""
Oct 30 18:59:02.034217 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.034179    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-home\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.034217 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.034219    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-tmp-dir\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.034503 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.034247    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355c74b1-f4af-41d2-a557-93b3e62f7499-tls-certs\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.034563 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.034501    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7x7\" (UniqueName: \"kubernetes.io/projected/355c74b1-f4af-41d2-a557-93b3e62f7499-kube-api-access-9c7x7\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.034623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.034580    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-model-cache\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.034623 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.034610    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-dshm\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.060454 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.060415    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-66bdc0\"/\"kube-root-ca.crt\""
Oct 30 18:59:02.060628 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.060570    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-66bdc0\"/\"distinct-w2-66bdc0-kserve-self-signed-certs\""
Oct 30 18:59:02.060677 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.060631    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-66bdc0\"/\"default-dockercfg-qkspm\""
Oct 30 18:59:02.063766 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.063743    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"]
Oct 30 18:59:02.135691 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.135656    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355c74b1-f4af-41d2-a557-93b3e62f7499-tls-certs\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.135892 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.135807    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7x7\" (UniqueName: \"kubernetes.io/projected/355c74b1-f4af-41d2-a557-93b3e62f7499-kube-api-access-9c7x7\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.135892 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.135869    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-model-cache\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.135969 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.135894    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-dshm\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.136007 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.135974    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-home\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.136007 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.135998    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-tmp-dir\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.136420 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.136393    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-model-cache\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.136420 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.136419    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-tmp-dir\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.136565 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.136453    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-home\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.138571 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.138545    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-dshm\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.138830 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.138814    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355c74b1-f4af-41d2-a557-93b3e62f7499-tls-certs\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.194013 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.193978    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7x7\" (UniqueName: \"kubernetes.io/projected/355c74b1-f4af-41d2-a557-93b3e62f7499-kube-api-access-9c7x7\") pod \"distinct-w2-66bdc0-kserve-7f89597b4b-p28qx\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.269265 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.269132    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:02.663058 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:02.663031    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"]
Oct 30 18:59:02.664383 ip-10-0-143-97 kubenswrapper[6014]: W1030 18:59:02.664350    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod355c74b1_f4af_41d2_a557_93b3e62f7499.slice/crio-29e82832f224ccff3e32b3867442acc960a8a88ea07842766a8e26f3826f2491 WatchSource:0}: Error finding container 29e82832f224ccff3e32b3867442acc960a8a88ea07842766a8e26f3826f2491: Status 404 returned error can't find the container with id 29e82832f224ccff3e32b3867442acc960a8a88ea07842766a8e26f3826f2491
Oct 30 18:59:03.270969 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:03.270919    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" event={"ID":"355c74b1-f4af-41d2-a557-93b3e62f7499","Type":"ContainerStarted","Data":"174b6520e0e48214bf8fcf5514bc093982c7ead3368ca3beb042f0c045dd2e0d"}
Oct 30 18:59:03.270969 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:03.270976    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" event={"ID":"355c74b1-f4af-41d2-a557-93b3e62f7499","Type":"ContainerStarted","Data":"29e82832f224ccff3e32b3867442acc960a8a88ea07842766a8e26f3826f2491"}
Oct 30 18:59:03.387933 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:03.387882    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podStartSLOduration=2.38786903 podStartE2EDuration="2.38786903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:03.386133463 +0000 UTC m=+1182.029301077" watchObservedRunningTime="2026-10-30 18:59:03.38786903 +0000 UTC m=+1182.031036643"
Oct 30 18:59:04.269585 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:04.269535    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:12.269649 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:12.269611    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:12.283586 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:12.283553    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:12.342228 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:12.342194    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 18:59:20.166849 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:20.166815    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 18:59:23.261208 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:59:23.261172    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 18:59:33.943297 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:33.943262    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"]
Oct 30 18:59:33.947471 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:33.947448    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.050292 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.050247    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-41db7c\"/\"kube-root-ca.crt\""
Oct 30 18:59:34.050292 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.050244    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-41db7c\"/\"openshift-service-ca.crt\""
Oct 30 18:59:34.050621 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.050602    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-41db7c\"/\"default-dockercfg-46k5f\""
Oct 30 18:59:34.050698 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.050606    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-41db7c\"/\"distinct-2-w6-41db7c-kserve-self-signed-certs\""
Oct 30 18:59:34.053996 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.053975    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"]
Oct 30 18:59:34.068203 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.068179    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-home\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.068327 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.068212    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-model-cache\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.068327 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.068238    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsvgk\" (UniqueName: \"kubernetes.io/projected/799cb959-41de-49dd-bab4-d64a2d054208-kube-api-access-jsvgk\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.068409 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.068340    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799cb959-41de-49dd-bab4-d64a2d054208-tls-certs\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.068458 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.068444    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-dshm\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.068602 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.068583    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-tmp-dir\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.170335 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.170298    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvgk\" (UniqueName: \"kubernetes.io/projected/799cb959-41de-49dd-bab4-d64a2d054208-kube-api-access-jsvgk\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.170513 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.170352    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799cb959-41de-49dd-bab4-d64a2d054208-tls-certs\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.170513 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.170431    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-dshm\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.170591 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.170516    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-tmp-dir\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.170591 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.170574    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-home\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.170664 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.170590    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-model-cache\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.170916 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.170893    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-tmp-dir\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.171036 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.170970    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-model-cache\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.171097 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.171040    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-home\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.172978 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.172947    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-dshm\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.173107 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.173091    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799cb959-41de-49dd-bab4-d64a2d054208-tls-certs\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.227737 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.227664    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsvgk\" (UniqueName: \"kubernetes.io/projected/799cb959-41de-49dd-bab4-d64a2d054208-kube-api-access-jsvgk\") pod \"distinct-2-w6-41db7c-kserve-84f956764c-wmb2r\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") " pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.260479 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.260444    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:34.639013 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.638985    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"]
Oct 30 18:59:34.662669 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:34.662639    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" event={"ID":"799cb959-41de-49dd-bab4-d64a2d054208","Type":"ContainerStarted","Data":"a59289b25e1a44eb83aa83c8a8eea6bb8c8eda165eb7b4e986642d7963780ea3"}
Oct 30 18:59:35.671616 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:35.671572    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" event={"ID":"799cb959-41de-49dd-bab4-d64a2d054208","Type":"ContainerStarted","Data":"e3ff3c2e842a68a60eeff2c2e04e0a71298f949ae3aa587dc2730fd64e49d5a8"}
Oct 30 18:59:35.787953 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:35.787898    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podStartSLOduration=2.787882838 podStartE2EDuration="2.787882838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:35.785958312 +0000 UTC m=+1214.429125927" watchObservedRunningTime="2026-10-30 18:59:35.787882838 +0000 UTC m=+1214.431050452"
Oct 30 18:59:36.260646 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:36.260615    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:44.261590 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:44.261551    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:44.274493 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:44.274465    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:44.755404 ip-10-0-143-97 kubenswrapper[6014]: I1030 18:59:44.755377    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 18:59:53.257928 ip-10-0-143-97 kubenswrapper[6014]: E1030 18:59:53.257895    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:00:00.162945 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.162911    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"]
Oct 30 19:00:00.167410 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.167392    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"]
Oct 30 19:00:00.167545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.167527    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"
Oct 30 19:00:00.173841 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.172927    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"
Oct 30 19:00:00.189841 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.189795    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"]
Oct 30 19:00:00.194271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.194252    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"
Oct 30 19:00:00.212321 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.212297    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"]
Oct 30 19:00:00.216396 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.216380    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"
Oct 30 19:00:00.219408 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.219386    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:00:00.266841 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.266771    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"]
Oct 30 19:00:00.270552 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.270522    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"]
Oct 30 19:00:00.316940 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.316370    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"]
Oct 30 19:00:00.320649 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.320621    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"]
Oct 30 19:00:00.343578 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.343544    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqds5\" (UniqueName: \"kubernetes.io/projected/5f801899-0f12-4a1f-941d-de0d95106d93-kube-api-access-jqds5\") pod \"maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v\" (UID: \"5f801899-0f12-4a1f-941d-de0d95106d93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"
Oct 30 19:00:00.343743 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.343600    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c\" (UID: \"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"
Oct 30 19:00:00.343796 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.343775    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7knv\" (UniqueName: \"kubernetes.io/projected/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-kube-api-access-v7knv\") pod \"maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c\" (UID: \"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"
Oct 30 19:00:00.343838 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.343825    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv5br\" (UniqueName: \"kubernetes.io/projected/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-kube-api-access-mv5br\") pod \"maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv\" (UID: \"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"
Oct 30 19:00:00.344056 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.344035    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwgn2\" (UniqueName: \"kubernetes.io/projected/7a647fbc-736b-4ea5-9375-fffb58047e03-kube-api-access-zwgn2\") pod \"maas-api-key-cleanup-29889780-gxfj2\" (UID: \"7a647fbc-736b-4ea5-9375-fffb58047e03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"
Oct 30 19:00:00.344101 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.344071    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv\" (UID: \"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"
Oct 30 19:00:00.344183 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.344106    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f801899-0f12-4a1f-941d-de0d95106d93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v\" (UID: \"5f801899-0f12-4a1f-941d-de0d95106d93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"
Oct 30 19:00:00.344183 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.344165    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a647fbc-736b-4ea5-9375-fffb58047e03-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-gxfj2\" (UID: \"7a647fbc-736b-4ea5-9375-fffb58047e03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"
Oct 30 19:00:00.446262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.446174    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqds5\" (UniqueName: \"kubernetes.io/projected/5f801899-0f12-4a1f-941d-de0d95106d93-kube-api-access-jqds5\") pod \"maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v\" (UID: \"5f801899-0f12-4a1f-941d-de0d95106d93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"
Oct 30 19:00:00.446262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.446241    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c\" (UID: \"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"
Oct 30 19:00:00.446482 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.446294    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7knv\" (UniqueName: \"kubernetes.io/projected/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-kube-api-access-v7knv\") pod \"maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c\" (UID: \"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"
Oct 30 19:00:00.446482 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.446329    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5br\" (UniqueName: \"kubernetes.io/projected/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-kube-api-access-mv5br\") pod \"maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv\" (UID: \"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"
Oct 30 19:00:00.446611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.446487    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwgn2\" (UniqueName: \"kubernetes.io/projected/7a647fbc-736b-4ea5-9375-fffb58047e03-kube-api-access-zwgn2\") pod \"maas-api-key-cleanup-29889780-gxfj2\" (UID: \"7a647fbc-736b-4ea5-9375-fffb58047e03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"
Oct 30 19:00:00.446611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.446511    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv\" (UID: \"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"
Oct 30 19:00:00.446611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.446547    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f801899-0f12-4a1f-941d-de0d95106d93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v\" (UID: \"5f801899-0f12-4a1f-941d-de0d95106d93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"
Oct 30 19:00:00.446611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.446577    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a647fbc-736b-4ea5-9375-fffb58047e03-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-gxfj2\" (UID: \"7a647fbc-736b-4ea5-9375-fffb58047e03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"
Oct 30 19:00:00.447038 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.447011    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c\" (UID: \"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"
Oct 30 19:00:00.447229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.447213    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv\" (UID: \"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"
Oct 30 19:00:00.447341 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.447314    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f801899-0f12-4a1f-941d-de0d95106d93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v\" (UID: \"5f801899-0f12-4a1f-941d-de0d95106d93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"
Oct 30 19:00:00.447460 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.447427    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a647fbc-736b-4ea5-9375-fffb58047e03-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-gxfj2\" (UID: \"7a647fbc-736b-4ea5-9375-fffb58047e03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"
Oct 30 19:00:00.508784 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.508742    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7knv\" (UniqueName: \"kubernetes.io/projected/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-kube-api-access-v7knv\") pod \"maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c\" (UID: \"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"
Oct 30 19:00:00.509349 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.509324    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwgn2\" (UniqueName: \"kubernetes.io/projected/7a647fbc-736b-4ea5-9375-fffb58047e03-kube-api-access-zwgn2\") pod \"maas-api-key-cleanup-29889780-gxfj2\" (UID: \"7a647fbc-736b-4ea5-9375-fffb58047e03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"
Oct 30 19:00:00.509487 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.509350    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqds5\" (UniqueName: \"kubernetes.io/projected/5f801899-0f12-4a1f-941d-de0d95106d93-kube-api-access-jqds5\") pod \"maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v\" (UID: \"5f801899-0f12-4a1f-941d-de0d95106d93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"
Oct 30 19:00:00.510018 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.509999    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv5br\" (UniqueName: \"kubernetes.io/projected/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-kube-api-access-mv5br\") pod \"maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv\" (UID: \"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"
Oct 30 19:00:00.530240 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.530207    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"
Oct 30 19:00:00.792009 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.791918    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"
Oct 30 19:00:00.798352 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.798327    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"
Oct 30 19:00:00.808412 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.808383    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"
Oct 30 19:00:00.912195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:00.911019    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"]
Oct 30 19:00:00.913456 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:00.913423    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a647fbc_736b_4ea5_9375_fffb58047e03.slice/crio-d5f95e0bc4f923694ddd5ced36d8fc3539a6001ec3736a3ef9a9a7c2bc3e0e05 WatchSource:0}: Error finding container d5f95e0bc4f923694ddd5ced36d8fc3539a6001ec3736a3ef9a9a7c2bc3e0e05: Status 404 returned error can't find the container with id d5f95e0bc4f923694ddd5ced36d8fc3539a6001ec3736a3ef9a9a7c2bc3e0e05
Oct 30 19:00:01.189177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.189128    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"]
Oct 30 19:00:01.191996 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:01.191965    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca4a6f1_c2b4_4285_b19e_3cbb46cf13a9.slice/crio-b21bc46ddd357cdf69b752fd04266cd44d5b9e20a6e0084c6975740968563e11 WatchSource:0}: Error finding container b21bc46ddd357cdf69b752fd04266cd44d5b9e20a6e0084c6975740968563e11: Status 404 returned error can't find the container with id b21bc46ddd357cdf69b752fd04266cd44d5b9e20a6e0084c6975740968563e11
Oct 30 19:00:01.195427 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.195399    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"]
Oct 30 19:00:01.196631 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:01.196610    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f801899_0f12_4a1f_941d_de0d95106d93.slice/crio-997aacfd7b3fc12ee348efb8b121ec673af64e005732d424c4a0d45bb05f0b44 WatchSource:0}: Error finding container 997aacfd7b3fc12ee348efb8b121ec673af64e005732d424c4a0d45bb05f0b44: Status 404 returned error can't find the container with id 997aacfd7b3fc12ee348efb8b121ec673af64e005732d424c4a0d45bb05f0b44
Oct 30 19:00:01.238919 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.238890    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"]
Oct 30 19:00:01.239705 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:01.239681    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28310ee_7c45_4a7f_bd1b_c6db37fd4a64.slice/crio-05c3731822999382e9263cd5e1e4b73132f6160f5f3b5ccf167eeadb0a639cc7 WatchSource:0}: Error finding container 05c3731822999382e9263cd5e1e4b73132f6160f5f3b5ccf167eeadb0a639cc7: Status 404 returned error can't find the container with id 05c3731822999382e9263cd5e1e4b73132f6160f5f3b5ccf167eeadb0a639cc7
Oct 30 19:00:01.882800 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.882765    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv" event={"ID":"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64","Type":"ContainerStarted","Data":"f8d767e745b2c8612a210ca0b388e4715f5c05a193adf67866eb3568c610798c"}
Oct 30 19:00:01.882995 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.882825    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv" event={"ID":"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64","Type":"ContainerStarted","Data":"05c3731822999382e9263cd5e1e4b73132f6160f5f3b5ccf167eeadb0a639cc7"}
Oct 30 19:00:01.884714 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.884682    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c" event={"ID":"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9","Type":"ContainerStarted","Data":"c556f4b22d344ad21a4ce61fbba3d57f9596bf394e0034a43f2858737835e7e3"}
Oct 30 19:00:01.884854 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.884739    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c" event={"ID":"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9","Type":"ContainerStarted","Data":"b21bc46ddd357cdf69b752fd04266cd44d5b9e20a6e0084c6975740968563e11"}
Oct 30 19:00:01.886627 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.886599    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v" event={"ID":"5f801899-0f12-4a1f-941d-de0d95106d93","Type":"ContainerStarted","Data":"a5bdaba78c391d3ff1ee1223ffc769feab2b9baa20d60b1fd8e15fc5c855d48e"}
Oct 30 19:00:01.886755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.886648    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v" event={"ID":"5f801899-0f12-4a1f-941d-de0d95106d93","Type":"ContainerStarted","Data":"997aacfd7b3fc12ee348efb8b121ec673af64e005732d424c4a0d45bb05f0b44"}
Oct 30 19:00:01.888296 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.888268    6014 generic.go:356] "Generic (PLEG): container finished" podID="7a647fbc-736b-4ea5-9375-fffb58047e03" containerID="9fc8c624a6916b783955864caa540be702c57ade727122946e76632692757948" exitCode=60
Oct 30 19:00:01.888402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.888301    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2" event={"ID":"7a647fbc-736b-4ea5-9375-fffb58047e03","Type":"ContainerDied","Data":"9fc8c624a6916b783955864caa540be702c57ade727122946e76632692757948"}
Oct 30 19:00:01.888402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.888343    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2" event={"ID":"7a647fbc-736b-4ea5-9375-fffb58047e03","Type":"ContainerStarted","Data":"d5f95e0bc4f923694ddd5ced36d8fc3539a6001ec3736a3ef9a9a7c2bc3e0e05"}
Oct 30 19:00:01.888517 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.888504    6014 scope.go:122] "RemoveContainer" containerID="9fc8c624a6916b783955864caa540be702c57ade727122946e76632692757948"
Oct 30 19:00:01.995972 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:01.995920    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv" podStartSLOduration=1.995904431 podStartE2EDuration="1.995904431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:01.994131198 +0000 UTC m=+1240.637298813" watchObservedRunningTime="2026-10-30 19:00:01.995904431 +0000 UTC m=+1240.639072045"
Oct 30 19:00:02.220382 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.220210    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c" podStartSLOduration=2.22019519 podStartE2EDuration="2.22019519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.2193873 +0000 UTC m=+1240.862554913" watchObservedRunningTime="2026-10-30 19:00:02.22019519 +0000 UTC m=+1240.863362803"
Oct 30 19:00:02.335768 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.335715    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v" podStartSLOduration=2.335700621 podStartE2EDuration="2.335700621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.332037221 +0000 UTC m=+1240.975204834" watchObservedRunningTime="2026-10-30 19:00:02.335700621 +0000 UTC m=+1240.978868235"
Oct 30 19:00:02.897785 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.897751    6014 generic.go:356] "Generic (PLEG): container finished" podID="7a647fbc-736b-4ea5-9375-fffb58047e03" containerID="199e4c1f020d20b3f83c21407047079bc5b74ebc0da0512a9c818521cf512d8d" exitCode=60
Oct 30 19:00:02.897975 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.897796    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2" event={"ID":"7a647fbc-736b-4ea5-9375-fffb58047e03","Type":"ContainerDied","Data":"199e4c1f020d20b3f83c21407047079bc5b74ebc0da0512a9c818521cf512d8d"}
Oct 30 19:00:02.897975 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.897883    6014 scope.go:122] "RemoveContainer" containerID="9fc8c624a6916b783955864caa540be702c57ade727122946e76632692757948"
Oct 30 19:00:02.898203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.898188    6014 scope.go:122] "RemoveContainer" containerID="199e4c1f020d20b3f83c21407047079bc5b74ebc0da0512a9c818521cf512d8d"
Oct 30 19:00:02.898531 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:02.898506    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-gxfj2_odh-ai-gateway-infra(7a647fbc-736b-4ea5-9375-fffb58047e03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2" podUID="7a647fbc-736b-4ea5-9375-fffb58047e03"
Oct 30 19:00:02.899920 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.899898    6014 generic.go:356] "Generic (PLEG): container finished" podID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" containerID="f8d767e745b2c8612a210ca0b388e4715f5c05a193adf67866eb3568c610798c" exitCode=60
Oct 30 19:00:02.900006 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.899989    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv" event={"ID":"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64","Type":"ContainerDied","Data":"f8d767e745b2c8612a210ca0b388e4715f5c05a193adf67866eb3568c610798c"}
Oct 30 19:00:02.900296 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.900274    6014 scope.go:122] "RemoveContainer" containerID="f8d767e745b2c8612a210ca0b388e4715f5c05a193adf67866eb3568c610798c"
Oct 30 19:00:02.901684 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.901662    6014 generic.go:356] "Generic (PLEG): container finished" podID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" containerID="c556f4b22d344ad21a4ce61fbba3d57f9596bf394e0034a43f2858737835e7e3" exitCode=60
Oct 30 19:00:02.901784 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.901734    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c" event={"ID":"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9","Type":"ContainerDied","Data":"c556f4b22d344ad21a4ce61fbba3d57f9596bf394e0034a43f2858737835e7e3"}
Oct 30 19:00:02.902040 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.902021    6014 scope.go:122] "RemoveContainer" containerID="c556f4b22d344ad21a4ce61fbba3d57f9596bf394e0034a43f2858737835e7e3"
Oct 30 19:00:02.904379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.904348    6014 generic.go:356] "Generic (PLEG): container finished" podID="5f801899-0f12-4a1f-941d-de0d95106d93" containerID="a5bdaba78c391d3ff1ee1223ffc769feab2b9baa20d60b1fd8e15fc5c855d48e" exitCode=60
Oct 30 19:00:02.904494 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.904480    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v" event={"ID":"5f801899-0f12-4a1f-941d-de0d95106d93","Type":"ContainerDied","Data":"a5bdaba78c391d3ff1ee1223ffc769feab2b9baa20d60b1fd8e15fc5c855d48e"}
Oct 30 19:00:02.904783 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:02.904767    6014 scope.go:122] "RemoveContainer" containerID="a5bdaba78c391d3ff1ee1223ffc769feab2b9baa20d60b1fd8e15fc5c855d48e"
Oct 30 19:00:03.548843 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.548780    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"]
Oct 30 19:00:03.557618 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.555244    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.613206 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.613172    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"]
Oct 30 19:00:03.693226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.693187    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hsnd\" (UniqueName: \"kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-kube-api-access-6hsnd\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.693441 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.693295    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.693441 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.693328    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d328b8d4-3583-4984-9435-1cb4349ae6aa-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.693441 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.693407    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.693441 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.693436    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.693654 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.693512    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.693654 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.693581    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.693757 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.693676    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.693757 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.693731    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.795447 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.795409    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.795447 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.795452    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d328b8d4-3583-4984-9435-1cb4349ae6aa-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.795695 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.795514    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.795695 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.795534    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.795695 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.795595    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.795695 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.795651    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.795881 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.795708    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.795881 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.795743    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.795994 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.795947    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hsnd\" (UniqueName: \"kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-kube-api-access-6hsnd\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.796124 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.796103    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.796263 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.796196    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.796263 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.796231    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.796460 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.796442    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.796606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.796584    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d328b8d4-3583-4984-9435-1cb4349ae6aa-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.798262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.798240    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.798262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.798256    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.855803 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.855770    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.855959 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.855839    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hsnd\" (UniqueName: \"kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-kube-api-access-6hsnd\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.879182 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.878456    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:03.916079 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.916050    6014 scope.go:122] "RemoveContainer" containerID="199e4c1f020d20b3f83c21407047079bc5b74ebc0da0512a9c818521cf512d8d"
Oct 30 19:00:03.916730 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:03.916457    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-gxfj2_odh-ai-gateway-infra(7a647fbc-736b-4ea5-9375-fffb58047e03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2" podUID="7a647fbc-736b-4ea5-9375-fffb58047e03"
Oct 30 19:00:03.917704 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.917681    6014 generic.go:356] "Generic (PLEG): container finished" podID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" containerID="93b7323600c1492a49ab9bdab6fb1990cdbadbcfa9bbf1207c0caaf849a70159" exitCode=60
Oct 30 19:00:03.917791 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.917782    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv" event={"ID":"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64","Type":"ContainerDied","Data":"93b7323600c1492a49ab9bdab6fb1990cdbadbcfa9bbf1207c0caaf849a70159"}
Oct 30 19:00:03.917831 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.917821    6014 scope.go:122] "RemoveContainer" containerID="f8d767e745b2c8612a210ca0b388e4715f5c05a193adf67866eb3568c610798c"
Oct 30 19:00:03.918262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.918243    6014 scope.go:122] "RemoveContainer" containerID="93b7323600c1492a49ab9bdab6fb1990cdbadbcfa9bbf1207c0caaf849a70159"
Oct 30 19:00:03.918570 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:03.918547    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv_odh-ai-gateway-infra(d28310ee-7c45-4a7f-bd1b-c6db37fd4a64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv" podUID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64"
Oct 30 19:00:03.921609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.921585    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c" event={"ID":"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9","Type":"ContainerStarted","Data":"44abdb8f3b80ada92be73cc78da846b80c64623da33a946843e6878482c48c41"}
Oct 30 19:00:03.924047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:03.924020    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v" event={"ID":"5f801899-0f12-4a1f-941d-de0d95106d93","Type":"ContainerStarted","Data":"5e47390adbd345acdbe39dc10b7f062f98158f7c2a73deee162bef2948866c10"}
Oct 30 19:00:04.058761 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.058733    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"]
Oct 30 19:00:04.062584 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.062555    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a16b51-txbrl"]
Oct 30 19:00:04.325373 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.325350    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"]
Oct 30 19:00:04.326421 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:04.326394    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd328b8d4_3583_4984_9435_1cb4349ae6aa.slice/crio-4225eb29b45a57f51df064d8ee74ec91d5ae1e43c0f8d2814c73e748bf0b8ca3 WatchSource:0}: Error finding container 4225eb29b45a57f51df064d8ee74ec91d5ae1e43c0f8d2814c73e748bf0b8ca3: Status 404 returned error can't find the container with id 4225eb29b45a57f51df064d8ee74ec91d5ae1e43c0f8d2814c73e748bf0b8ca3
Oct 30 19:00:04.328866 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.328836    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:04.328922 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.328899    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:04.328970 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.328928    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:04.851458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.851421    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"]
Oct 30 19:00:04.856969 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.856944    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:04.909928 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.909892    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:04.910255 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.910027    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:04.911464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.911435    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:04.911609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.911538    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:04.911609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.911567    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:04.911784 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.911755    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:04.911853 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.911835    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:04.911932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.911910    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:04.912084 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.911967    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msphp\" (UniqueName: \"kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-kube-api-access-msphp\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:04.913986 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.913964    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"]
Oct 30 19:00:04.950448 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.950416    6014 scope.go:122] "RemoveContainer" containerID="93b7323600c1492a49ab9bdab6fb1990cdbadbcfa9bbf1207c0caaf849a70159"
Oct 30 19:00:04.950883 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:04.950850    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv_odh-ai-gateway-infra(d28310ee-7c45-4a7f-bd1b-c6db37fd4a64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv" podUID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64"
Oct 30 19:00:04.952131 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.952105    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g" event={"ID":"d328b8d4-3583-4984-9435-1cb4349ae6aa","Type":"ContainerStarted","Data":"3c25dc92195ebeb056395c6e2d42da5031f7bf7c36f8af5e53888eac34d29280"}
Oct 30 19:00:04.952276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.952192    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g" event={"ID":"d328b8d4-3583-4984-9435-1cb4349ae6aa","Type":"ContainerStarted","Data":"4225eb29b45a57f51df064d8ee74ec91d5ae1e43c0f8d2814c73e748bf0b8ca3"}
Oct 30 19:00:04.954589 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.954566    6014 generic.go:356] "Generic (PLEG): container finished" podID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" containerID="44abdb8f3b80ada92be73cc78da846b80c64623da33a946843e6878482c48c41" exitCode=60
Oct 30 19:00:04.954703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.954673    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c" event={"ID":"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9","Type":"ContainerDied","Data":"44abdb8f3b80ada92be73cc78da846b80c64623da33a946843e6878482c48c41"}
Oct 30 19:00:04.954772 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.954750    6014 scope.go:122] "RemoveContainer" containerID="c556f4b22d344ad21a4ce61fbba3d57f9596bf394e0034a43f2858737835e7e3"
Oct 30 19:00:04.955671 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.955453    6014 scope.go:122] "RemoveContainer" containerID="44abdb8f3b80ada92be73cc78da846b80c64623da33a946843e6878482c48c41"
Oct 30 19:00:04.956196 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:04.955760    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c_odh-ai-gateway-infra(bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c" podUID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9"
Oct 30 19:00:04.956810 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.956781    6014 generic.go:356] "Generic (PLEG): container finished" podID="5f801899-0f12-4a1f-941d-de0d95106d93" containerID="5e47390adbd345acdbe39dc10b7f062f98158f7c2a73deee162bef2948866c10" exitCode=60
Oct 30 19:00:04.956899 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.956866    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v" event={"ID":"5f801899-0f12-4a1f-941d-de0d95106d93","Type":"ContainerDied","Data":"5e47390adbd345acdbe39dc10b7f062f98158f7c2a73deee162bef2948866c10"}
Oct 30 19:00:04.957042 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.957030    6014 scope.go:122] "RemoveContainer" containerID="5e47390adbd345acdbe39dc10b7f062f98158f7c2a73deee162bef2948866c10"
Oct 30 19:00:04.957288 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:04.957272    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v_odh-ai-gateway-infra(5f801899-0f12-4a1f-941d-de0d95106d93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v" podUID="5f801899-0f12-4a1f-941d-de0d95106d93"
Oct 30 19:00:04.967861 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:04.967837    6014 scope.go:122] "RemoveContainer" containerID="a5bdaba78c391d3ff1ee1223ffc769feab2b9baa20d60b1fd8e15fc5c855d48e"
Oct 30 19:00:05.013517 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.013488    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.013720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.013574    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.013720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.013617    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.013720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.013679    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.013720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.013696    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.014033 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.013841    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.014033 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.013906    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.014033 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.013940    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.014033 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.013959    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msphp\" (UniqueName: \"kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-kube-api-access-msphp\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.014351 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.014110    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.014351 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.014258    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.014731 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.014697    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.015307 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.015284    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.015396 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.015359    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.016583 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.016565    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.016813 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.016794    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.072792 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.072758    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.072978 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.072947    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msphp\" (UniqueName: \"kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-kube-api-access-msphp\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.119720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.119688    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"]
Oct 30 19:00:05.125953 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.125929    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.172508 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.172460    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:05.184000 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.183961    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"]
Oct 30 19:00:05.216736 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.216672    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.216736 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.216738    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lwwz\" (UniqueName: \"kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-kube-api-access-2lwwz\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.217022 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.216846    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.217022 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.216894    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.217022 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.216936    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.217177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.217075    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.217177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.217104    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.217177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.217152    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.217301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.217196    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.319394 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.319356    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.319588 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.319431    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.319588 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.319485    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.319669 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.319633    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.319704 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.319661    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.319742 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.319702    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.319776 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.319753    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.319836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.319818    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.319893 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.319876    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lwwz\" (UniqueName: \"kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-kube-api-access-2lwwz\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.320728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.320060    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.320728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.320131    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.320728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.319817    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.320728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.320331    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.320728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.320688    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.322653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.322625    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.322830 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.322795    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.376833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.376790    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.381661 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.381629    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lwwz\" (UniqueName: \"kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-kube-api-access-2lwwz\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.434643 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.434590    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g" podStartSLOduration=2.434574584 podStartE2EDuration="2.434574584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:05.43379825 +0000 UTC m=+1244.076965864" watchObservedRunningTime="2026-10-30 19:00:05.434574584 +0000 UTC m=+1244.077742198"
Oct 30 19:00:05.442372 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.442339    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:05.482344 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.482318    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774d9002-4d61-4dba-a869-3ef9353d170a" path="/var/lib/kubelet/pods/774d9002-4d61-4dba-a869-3ef9353d170a/volumes"
Oct 30 19:00:05.677274 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.677244    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"]
Oct 30 19:00:05.677384 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:05.677292    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd3e34f_96e3_4fee_9e15_4b884c33b0c0.slice/crio-6599d2c4981116a01a9f2e54ddfe6e8e808acc86b3718a064addcd9e86d360d8 WatchSource:0}: Error finding container 6599d2c4981116a01a9f2e54ddfe6e8e808acc86b3718a064addcd9e86d360d8: Status 404 returned error can't find the container with id 6599d2c4981116a01a9f2e54ddfe6e8e808acc86b3718a064addcd9e86d360d8
Oct 30 19:00:05.679928 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.679896    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:05.680008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.679958    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:05.680008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.679987    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:05.878207 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.870045    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"]
Oct 30 19:00:05.879492 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.879446    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:05.881860 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.881826    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:00:05.882082 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.882039    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g" podUID="d328b8d4-3583-4984-9435-1cb4349ae6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 19:00:05.882567 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:05.882524    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2c9ad2_ed90_4a65_adcf_3b2540f128ed.slice/crio-bc6d85dd84d6b03e670f3f79d9de4acf02d8fa47594dc3df7388c5950940647e WatchSource:0}: Error finding container bc6d85dd84d6b03e670f3f79d9de4acf02d8fa47594dc3df7388c5950940647e: Status 404 returned error can't find the container with id bc6d85dd84d6b03e670f3f79d9de4acf02d8fa47594dc3df7388c5950940647e
Oct 30 19:00:05.885300 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.885267    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:05.885399 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.885356    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:05.885467 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.885401    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:05.983124 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.983094    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" event={"ID":"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0","Type":"ContainerStarted","Data":"4640b3770dffad6254fb51f6b4ac62d506f1d92c9e5f105bdc00d2145aa8e026"}
Oct 30 19:00:05.983337 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.983318    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" event={"ID":"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0","Type":"ContainerStarted","Data":"6599d2c4981116a01a9f2e54ddfe6e8e808acc86b3718a064addcd9e86d360d8"}
Oct 30 19:00:05.986638 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.986264    6014 scope.go:122] "RemoveContainer" containerID="44abdb8f3b80ada92be73cc78da846b80c64623da33a946843e6878482c48c41"
Oct 30 19:00:05.986638 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:05.986587    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c_odh-ai-gateway-infra(bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c" podUID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9"
Oct 30 19:00:05.988540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.988511    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" event={"ID":"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed","Type":"ContainerStarted","Data":"bc6d85dd84d6b03e670f3f79d9de4acf02d8fa47594dc3df7388c5950940647e"}
Oct 30 19:00:05.990815 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:05.990790    6014 scope.go:122] "RemoveContainer" containerID="5e47390adbd345acdbe39dc10b7f062f98158f7c2a73deee162bef2948866c10"
Oct 30 19:00:05.991073 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:05.991054    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v_odh-ai-gateway-infra(5f801899-0f12-4a1f-941d-de0d95106d93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v" podUID="5f801899-0f12-4a1f-941d-de0d95106d93"
Oct 30 19:00:06.101383 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:06.101326    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" podStartSLOduration=2.101307937 podStartE2EDuration="2.101307937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:06.099886795 +0000 UTC m=+1244.743054410" watchObservedRunningTime="2026-10-30 19:00:06.101307937 +0000 UTC m=+1244.744475552"
Oct 30 19:00:06.173379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:06.173325    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:06.175571 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:06.175528    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:06.175734 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:06.175591    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:00:06.882131 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:06.881630    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:00:06.882131 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:06.881695    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g" podUID="d328b8d4-3583-4984-9435-1cb4349ae6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 19:00:07.029379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:07.029260    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" event={"ID":"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed","Type":"ContainerStarted","Data":"7dd0d4e3f6702f579db8bc8f143cd3162712cf8a74c267328e83f4e55f467192"}
Oct 30 19:00:07.199227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:07.196669    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:07.199227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:07.196733    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:00:07.199227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:07.198010    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" podStartSLOduration=2.197991024 podStartE2EDuration="2.197991024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:07.170828046 +0000 UTC m=+1245.813995660" watchObservedRunningTime="2026-10-30 19:00:07.197991024 +0000 UTC m=+1245.841158639"
Oct 30 19:00:07.442594 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:07.442556    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:07.444156 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:07.444113    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:07.444272 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:07.444182    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" podUID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:00:07.890282 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:07.890244    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:00:07.890888 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:07.890855    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g" podUID="d328b8d4-3583-4984-9435-1cb4349ae6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 19:00:08.181005 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:08.180737    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:08.181326 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:08.181275    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:00:08.443078 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:08.442987    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:08.443078 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:08.443045    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" podUID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:00:08.879633 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:08.879594    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:00:08.879893 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:08.879865    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g" podUID="d328b8d4-3583-4984-9435-1cb4349ae6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 19:00:09.176236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:09.176116    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:09.176814 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:09.176784    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:00:09.442852 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:09.442771    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:09.443074 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:09.443050    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" podUID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:00:09.885166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:09.881619    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:00:09.885166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:09.881684    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g" podUID="d328b8d4-3583-4984-9435-1cb4349ae6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 19:00:10.187228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:10.186607    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:10.187228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:10.186679    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:00:10.442898 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:10.442678    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:10.442898 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:10.442746    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" podUID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:00:10.895167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:10.894569    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:00:10.895167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:10.894632    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g" podUID="d328b8d4-3583-4984-9435-1cb4349ae6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 19:00:11.219007 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:11.218910    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:11.219799 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:11.219463    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:00:11.445589 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:11.445556    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:11.445847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:11.445823    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" podUID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:00:11.881166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:11.880779    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:00:11.881166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:11.880868    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g" podUID="d328b8d4-3583-4984-9435-1cb4349ae6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 19:00:12.183229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:12.174533    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:12.183229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:12.174595    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:00:12.443609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:12.443510    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:12.443609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:12.443582    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" podUID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:00:12.736179 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:12.736034    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:00:12.886450 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:12.886415    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:12.888032 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:12.888010    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:12.888838 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:12.888818    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:00:13.173754 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:13.173714    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:13.173934 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:13.173782    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:00:13.443542 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:13.443439    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:13.443542 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:13.443522    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" podUID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:00:14.241417 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:14.241363    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:14.242648 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:14.242623    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:14.243913 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:14.243891    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:00:14.454186 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:14.454056    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:14.455092 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:14.455070    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:14.455986 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:14.455958    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:00:15.479496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:15.479437    6014 scope.go:122] "RemoveContainer" containerID="44abdb8f3b80ada92be73cc78da846b80c64623da33a946843e6878482c48c41"
Oct 30 19:00:15.526445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:15.526405    6014 scope.go:122] "RemoveContainer" containerID="5e47390adbd345acdbe39dc10b7f062f98158f7c2a73deee162bef2948866c10"
Oct 30 19:00:16.047126 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:16.047040    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"]
Oct 30 19:00:16.067767 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:16.067742    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-856efad4-llmpt"]
Oct 30 19:00:16.334764 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:16.334721    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c" event={"ID":"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9","Type":"ContainerStarted","Data":"5f92d400da4d2d2b30425db5b77b76592a726211855beb850e8e3565597b3fbc"}
Oct 30 19:00:16.336799 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:16.336772    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v" event={"ID":"5f801899-0f12-4a1f-941d-de0d95106d93","Type":"ContainerStarted","Data":"a22a55efb89e4850bba8dd1ace0e200cf4a606e698bbd6684e17ebad4b8e6452"}
Oct 30 19:00:16.469567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:16.469539    6014 scope.go:122] "RemoveContainer" containerID="199e4c1f020d20b3f83c21407047079bc5b74ebc0da0512a9c818521cf512d8d"
Oct 30 19:00:17.346939 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.346907    6014 generic.go:356] "Generic (PLEG): container finished" podID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" containerID="5f92d400da4d2d2b30425db5b77b76592a726211855beb850e8e3565597b3fbc" exitCode=60
Oct 30 19:00:17.347520 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.346978    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c" event={"ID":"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9","Type":"ContainerDied","Data":"5f92d400da4d2d2b30425db5b77b76592a726211855beb850e8e3565597b3fbc"}
Oct 30 19:00:17.347520 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.347045    6014 scope.go:122] "RemoveContainer" containerID="44abdb8f3b80ada92be73cc78da846b80c64623da33a946843e6878482c48c41"
Oct 30 19:00:17.347646 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.347529    6014 scope.go:122] "RemoveContainer" containerID="5f92d400da4d2d2b30425db5b77b76592a726211855beb850e8e3565597b3fbc"
Oct 30 19:00:17.347919 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:17.347893    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c_odh-ai-gateway-infra(bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c" podUID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9"
Oct 30 19:00:17.349332 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.349310    6014 generic.go:356] "Generic (PLEG): container finished" podID="5f801899-0f12-4a1f-941d-de0d95106d93" containerID="a22a55efb89e4850bba8dd1ace0e200cf4a606e698bbd6684e17ebad4b8e6452" exitCode=60
Oct 30 19:00:17.349438 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.349385    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v" event={"ID":"5f801899-0f12-4a1f-941d-de0d95106d93","Type":"ContainerDied","Data":"a22a55efb89e4850bba8dd1ace0e200cf4a606e698bbd6684e17ebad4b8e6452"}
Oct 30 19:00:17.350123 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.349747    6014 scope.go:122] "RemoveContainer" containerID="a22a55efb89e4850bba8dd1ace0e200cf4a606e698bbd6684e17ebad4b8e6452"
Oct 30 19:00:17.350123 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:17.350097    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v_odh-ai-gateway-infra(5f801899-0f12-4a1f-941d-de0d95106d93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v" podUID="5f801899-0f12-4a1f-941d-de0d95106d93"
Oct 30 19:00:17.351817 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.351793    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2" event={"ID":"7a647fbc-736b-4ea5-9375-fffb58047e03","Type":"ContainerStarted","Data":"e9c208f1f9621c465d3ca99057f96c583290e639b7fcce975def96abe1db47c5"}
Oct 30 19:00:17.428898 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.428862    6014 scope.go:122] "RemoveContainer" containerID="5e47390adbd345acdbe39dc10b7f062f98158f7c2a73deee162bef2948866c10"
Oct 30 19:00:17.462620 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.462589    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"]
Oct 30 19:00:17.469501 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.469470    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"
Oct 30 19:00:17.481736 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.481709    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e81e058-5962-4007-a197-7407e64604c7" path="/var/lib/kubelet/pods/0e81e058-5962-4007-a197-7407e64604c7/volumes"
Oct 30 19:00:17.511517 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.511486    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"]
Oct 30 19:00:17.521896 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.521865    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:17.523781 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.523754    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"]
Oct 30 19:00:17.532234 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.532199    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aef5529e-bb5nj\" (UID: \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"
Oct 30 19:00:17.532550 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.532529    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aef5529e-bb5nj\" (UID: \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"
Oct 30 19:00:17.532645 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.532625    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7pt8\" (UniqueName: \"kubernetes.io/projected/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-kube-api-access-w7pt8\") pod \"maas-api-revoke-sub-aef5529e-bb5nj\" (UID: \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"
Oct 30 19:00:17.574248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.574210    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"]
Oct 30 19:00:17.634210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.634090    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aef5529e-bb5nj\" (UID: \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"
Oct 30 19:00:17.634210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.634200    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7pt8\" (UniqueName: \"kubernetes.io/projected/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-kube-api-access-w7pt8\") pod \"maas-api-revoke-sub-aef5529e-bb5nj\" (UID: \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"
Oct 30 19:00:17.634430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.634278    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aef5529e-bb5nj\" (UID: \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"
Oct 30 19:00:17.635043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.635019    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aef5529e-bb5nj\" (UID: \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"
Oct 30 19:00:17.636737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.636717    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aef5529e-bb5nj\" (UID: \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"
Oct 30 19:00:17.691033 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.690992    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7pt8\" (UniqueName: \"kubernetes.io/projected/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-kube-api-access-w7pt8\") pod \"maas-api-revoke-sub-aef5529e-bb5nj\" (UID: \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"
Oct 30 19:00:17.695882 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.695837    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2" podStartSLOduration=17.695823075 podStartE2EDuration="17.695823075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:17.694024954 +0000 UTC m=+1256.337192568" watchObservedRunningTime="2026-10-30 19:00:17.695823075 +0000 UTC m=+1256.338990690"
Oct 30 19:00:17.786995 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:17.786953    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"
Oct 30 19:00:18.175647 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.175621    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"]
Oct 30 19:00:18.176377 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:18.176346    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc903929a_5bc5_426e_9bcc_caa2fe0e0d2d.slice/crio-533efb72729e9c1622f4731eb6cae6240284a9dfa0f6e1169f60d86c1ad5ca3e WatchSource:0}: Error finding container 533efb72729e9c1622f4731eb6cae6240284a9dfa0f6e1169f60d86c1ad5ca3e: Status 404 returned error can't find the container with id 533efb72729e9c1622f4731eb6cae6240284a9dfa0f6e1169f60d86c1ad5ca3e
Oct 30 19:00:18.364876 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.364842    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj" event={"ID":"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d","Type":"ContainerStarted","Data":"533efb72729e9c1622f4731eb6cae6240284a9dfa0f6e1169f60d86c1ad5ca3e"}
Oct 30 19:00:18.371063 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.371036    6014 generic.go:356] "Generic (PLEG): container finished" podID="7a647fbc-736b-4ea5-9375-fffb58047e03" containerID="e9c208f1f9621c465d3ca99057f96c583290e639b7fcce975def96abe1db47c5" exitCode=60
Oct 30 19:00:18.371188 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.371113    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2" event={"ID":"7a647fbc-736b-4ea5-9375-fffb58047e03","Type":"ContainerDied","Data":"e9c208f1f9621c465d3ca99057f96c583290e639b7fcce975def96abe1db47c5"}
Oct 30 19:00:18.371270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.371213    6014 scope.go:122] "RemoveContainer" containerID="199e4c1f020d20b3f83c21407047079bc5b74ebc0da0512a9c818521cf512d8d"
Oct 30 19:00:18.371653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.371638    6014 scope.go:122] "RemoveContainer" containerID="e9c208f1f9621c465d3ca99057f96c583290e639b7fcce975def96abe1db47c5"
Oct 30 19:00:18.372010 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:18.371992    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-gxfj2_odh-ai-gateway-infra(7a647fbc-736b-4ea5-9375-fffb58047e03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2" podUID="7a647fbc-736b-4ea5-9375-fffb58047e03"
Oct 30 19:00:18.469323 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.469294    6014 scope.go:122] "RemoveContainer" containerID="93b7323600c1492a49ab9bdab6fb1990cdbadbcfa9bbf1207c0caaf849a70159"
Oct 30 19:00:18.525965 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.525922    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"
Oct 30 19:00:18.546596 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.546565    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f801899-0f12-4a1f-941d-de0d95106d93-kube-api-access-jqds5\" (UniqueName: \"kubernetes.io/projected/5f801899-0f12-4a1f-941d-de0d95106d93-kube-api-access-jqds5\") pod \"5f801899-0f12-4a1f-941d-de0d95106d93\" (UID: \"5f801899-0f12-4a1f-941d-de0d95106d93\") "
Oct 30 19:00:18.546737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.546691    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f801899-0f12-4a1f-941d-de0d95106d93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f801899-0f12-4a1f-941d-de0d95106d93-maas-api-service-ca\") pod \"5f801899-0f12-4a1f-941d-de0d95106d93\" (UID: \"5f801899-0f12-4a1f-941d-de0d95106d93\") "
Oct 30 19:00:18.547115 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.547088    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f801899-0f12-4a1f-941d-de0d95106d93-maas-api-service-ca" pod "5f801899-0f12-4a1f-941d-de0d95106d93" (UID: "5f801899-0f12-4a1f-941d-de0d95106d93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:18.547585 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.547556    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f801899-0f12-4a1f-941d-de0d95106d93-maas-api-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.550243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.550222    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f801899-0f12-4a1f-941d-de0d95106d93-kube-api-access-jqds5" pod "5f801899-0f12-4a1f-941d-de0d95106d93" (UID: "5f801899-0f12-4a1f-941d-de0d95106d93"). InnerVolumeSpecName "kube-api-access-jqds5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:18.648901 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.648866    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqds5\" (UniqueName: \"kubernetes.io/projected/5f801899-0f12-4a1f-941d-de0d95106d93-kube-api-access-jqds5\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.686852 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.686791    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"
Oct 30 19:00:18.706640 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.706604    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"]
Oct 30 19:00:18.749439 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.749409    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-maas-api-service-ca\") pod \"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9\" (UID: \"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9\") "
Oct 30 19:00:18.749635 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.749608    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-kube-api-access-v7knv\" (UniqueName: \"kubernetes.io/projected/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-kube-api-access-v7knv\") pod \"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9\" (UID: \"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9\") "
Oct 30 19:00:18.749834 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.749811    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-maas-api-service-ca" pod "bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" (UID: "bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:18.750176 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.750158    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-maas-api-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.751827 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.751804    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-kube-api-access-v7knv" pod "bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" (UID: "bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9"). InnerVolumeSpecName "kube-api-access-v7knv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:18.852313 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:18.852277    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7knv\" (UniqueName: \"kubernetes.io/projected/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9-kube-api-access-v7knv\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:19.383943 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.383911    6014 generic.go:356] "Generic (PLEG): container finished" podID="c903929a-5bc5-426e-9bcc-caa2fe0e0d2d" containerID="2396cee5c6e9abae742469b89e050a4e89bdffb7f4de99cd60c6d051d658e9d0" exitCode=0
Oct 30 19:00:19.384334 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.383992    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj" event={"ID":"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d","Type":"ContainerDied","Data":"2396cee5c6e9abae742469b89e050a4e89bdffb7f4de99cd60c6d051d658e9d0"}
Oct 30 19:00:19.386047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.386014    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv" event={"ID":"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64","Type":"ContainerStarted","Data":"08c4cc7ab8cdd744c8e520937b7441883de959081173d0cca298354060c3406e"}
Oct 30 19:00:19.387636 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.387618    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"
Oct 30 19:00:19.387745 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.387635    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c" event={"ID":"bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9","Type":"ContainerDied","Data":"b21bc46ddd357cdf69b752fd04266cd44d5b9e20a6e0084c6975740968563e11"}
Oct 30 19:00:19.387745 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.387688    6014 scope.go:122] "RemoveContainer" containerID="5f92d400da4d2d2b30425db5b77b76592a726211855beb850e8e3565597b3fbc"
Oct 30 19:00:19.389099 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.389073    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v" event={"ID":"5f801899-0f12-4a1f-941d-de0d95106d93","Type":"ContainerDied","Data":"997aacfd7b3fc12ee348efb8b121ec673af64e005732d424c4a0d45bb05f0b44"}
Oct 30 19:00:19.389301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.389119    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"
Oct 30 19:00:19.400458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.400435    6014 scope.go:122] "RemoveContainer" containerID="a22a55efb89e4850bba8dd1ace0e200cf4a606e698bbd6684e17ebad4b8e6452"
Oct 30 19:00:19.536445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.536419    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"
Oct 30 19:00:19.564499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.564469    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a647fbc-736b-4ea5-9375-fffb58047e03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a647fbc-736b-4ea5-9375-fffb58047e03-maas-api-service-ca\") pod \"7a647fbc-736b-4ea5-9375-fffb58047e03\" (UID: \"7a647fbc-736b-4ea5-9375-fffb58047e03\") "
Oct 30 19:00:19.564677 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.564639    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a647fbc-736b-4ea5-9375-fffb58047e03-kube-api-access-zwgn2\" (UniqueName: \"kubernetes.io/projected/7a647fbc-736b-4ea5-9375-fffb58047e03-kube-api-access-zwgn2\") pod \"7a647fbc-736b-4ea5-9375-fffb58047e03\" (UID: \"7a647fbc-736b-4ea5-9375-fffb58047e03\") "
Oct 30 19:00:19.564853 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.564825    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a647fbc-736b-4ea5-9375-fffb58047e03-maas-api-service-ca" pod "7a647fbc-736b-4ea5-9375-fffb58047e03" (UID: "7a647fbc-736b-4ea5-9375-fffb58047e03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:19.565225 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.565208    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a647fbc-736b-4ea5-9375-fffb58047e03-maas-api-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:19.566903 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.566884    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a647fbc-736b-4ea5-9375-fffb58047e03-kube-api-access-zwgn2" pod "7a647fbc-736b-4ea5-9375-fffb58047e03" (UID: "7a647fbc-736b-4ea5-9375-fffb58047e03"). InnerVolumeSpecName "kube-api-access-zwgn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:19.666330 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.666294    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwgn2\" (UniqueName: \"kubernetes.io/projected/7a647fbc-736b-4ea5-9375-fffb58047e03-kube-api-access-zwgn2\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:19.878158 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.878097    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"]
Oct 30 19:00:19.883643 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.883606    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66bdc0-29889780-qw27c"]
Oct 30 19:00:19.970456 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.970378    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"]
Oct 30 19:00:19.971042 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971029    6014 cpu_manager.go:414] "containerMap: removing container" podUID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" containerName="cleanup"
Oct 30 19:00:19.971081 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971044    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" containerName="cleanup"
Oct 30 19:00:19.971081 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971056    6014 cpu_manager.go:414] "containerMap: removing container" podUID="7a647fbc-736b-4ea5-9375-fffb58047e03" containerName="cleanup"
Oct 30 19:00:19.971081 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971062    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a647fbc-736b-4ea5-9375-fffb58047e03" containerName="cleanup"
Oct 30 19:00:19.971081 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971071    6014 cpu_manager.go:414] "containerMap: removing container" podUID="5f801899-0f12-4a1f-941d-de0d95106d93" containerName="cleanup"
Oct 30 19:00:19.971081 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971076    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f801899-0f12-4a1f-941d-de0d95106d93" containerName="cleanup"
Oct 30 19:00:19.971254 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971092    6014 cpu_manager.go:414] "containerMap: removing container" podUID="7a647fbc-736b-4ea5-9375-fffb58047e03" containerName="cleanup"
Oct 30 19:00:19.971254 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971097    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a647fbc-736b-4ea5-9375-fffb58047e03" containerName="cleanup"
Oct 30 19:00:19.971254 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971112    6014 cpu_manager.go:414] "containerMap: removing container" podUID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" containerName="cleanup"
Oct 30 19:00:19.971254 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971116    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" containerName="cleanup"
Oct 30 19:00:19.971254 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971127    6014 cpu_manager.go:414] "containerMap: removing container" podUID="7a647fbc-736b-4ea5-9375-fffb58047e03" containerName="cleanup"
Oct 30 19:00:19.971254 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971131    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a647fbc-736b-4ea5-9375-fffb58047e03" containerName="cleanup"
Oct 30 19:00:19.971254 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971154    6014 cpu_manager.go:414] "containerMap: removing container" podUID="5f801899-0f12-4a1f-941d-de0d95106d93" containerName="cleanup"
Oct 30 19:00:19.971254 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971159    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f801899-0f12-4a1f-941d-de0d95106d93" containerName="cleanup"
Oct 30 19:00:19.971486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971306    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a647fbc-736b-4ea5-9375-fffb58047e03" containerName="cleanup"
Oct 30 19:00:19.971486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971315    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" containerName="cleanup"
Oct 30 19:00:19.971486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971321    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a647fbc-736b-4ea5-9375-fffb58047e03" containerName="cleanup"
Oct 30 19:00:19.971486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971327    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f801899-0f12-4a1f-941d-de0d95106d93" containerName="cleanup"
Oct 30 19:00:19.971486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971334    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f801899-0f12-4a1f-941d-de0d95106d93" containerName="cleanup"
Oct 30 19:00:19.971486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971344    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" containerName="cleanup"
Oct 30 19:00:19.971486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.971349    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a647fbc-736b-4ea5-9375-fffb58047e03" containerName="cleanup"
Oct 30 19:00:19.975208 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.975187    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:19.999179 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:19.999129    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"]
Oct 30 19:00:20.008861 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.008828    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889780-fss9v"]
Oct 30 19:00:20.035969 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.035939    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"]
Oct 30 19:00:20.071958 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.071925    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.072116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.071979    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kplx8\" (UniqueName: \"kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-kube-api-access-kplx8\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.072116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.072056    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/898fcd26-1c1e-40e6-994d-619340f782cb-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.072286 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.072159    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.072286 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.072183    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.072286 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.072201    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/898fcd26-1c1e-40e6-994d-619340f782cb-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.072406 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.072311    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.072462 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.072437    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.072520 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.072501    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.173911 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.173872    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.174096 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.173932    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplx8\" (UniqueName: \"kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-kube-api-access-kplx8\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.174096 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.173957    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/898fcd26-1c1e-40e6-994d-619340f782cb-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.174096 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.174005    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.174096 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.174028    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.174402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.174189    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/898fcd26-1c1e-40e6-994d-619340f782cb-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.174402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.174250    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.174402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.174335    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.174402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.174394    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.174604 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.174504    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.174604 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.174541    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.174746 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.174724    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.174844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.174827    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.175285 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.175259    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/898fcd26-1c1e-40e6-994d-619340f782cb-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.176540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.176520    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.176765 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.176748    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/898fcd26-1c1e-40e6-994d-619340f782cb-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.236996 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.236836    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplx8\" (UniqueName: \"kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-kube-api-access-kplx8\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.243495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.243452    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.251308 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.251279    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"]
Oct 30 19:00:20.252034 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.252015    6014 cpu_manager.go:414] "containerMap: removing container" podUID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" containerName="cleanup"
Oct 30 19:00:20.252169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.252036    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" containerName="cleanup"
Oct 30 19:00:20.252169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.252057    6014 cpu_manager.go:414] "containerMap: removing container" podUID="5f801899-0f12-4a1f-941d-de0d95106d93" containerName="cleanup"
Oct 30 19:00:20.252169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.252065    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f801899-0f12-4a1f-941d-de0d95106d93" containerName="cleanup"
Oct 30 19:00:20.252334 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.252326    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" containerName="cleanup"
Oct 30 19:00:20.252384 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.252347    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f801899-0f12-4a1f-941d-de0d95106d93" containerName="cleanup"
Oct 30 19:00:20.257309 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.257288    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.289738 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.289704    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:20.314587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.314560    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"]
Oct 30 19:00:20.377070 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.377033    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.377237 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.377164    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q27x\" (UniqueName: \"kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-kube-api-access-7q27x\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.377237 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.377196    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.377237 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.377228    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.377420 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.377283    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.377420 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.377310    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.377420 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.377361    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.377568 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.377446    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.377568 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.377500    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.402539 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.402502    6014 generic.go:356] "Generic (PLEG): container finished" podID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" containerID="08c4cc7ab8cdd744c8e520937b7441883de959081173d0cca298354060c3406e" exitCode=60
Oct 30 19:00:20.403511 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.402577    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv" event={"ID":"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64","Type":"ContainerDied","Data":"08c4cc7ab8cdd744c8e520937b7441883de959081173d0cca298354060c3406e"}
Oct 30 19:00:20.403511 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.402651    6014 scope.go:122] "RemoveContainer" containerID="93b7323600c1492a49ab9bdab6fb1990cdbadbcfa9bbf1207c0caaf849a70159"
Oct 30 19:00:20.403511 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.403096    6014 scope.go:122] "RemoveContainer" containerID="08c4cc7ab8cdd744c8e520937b7441883de959081173d0cca298354060c3406e"
Oct 30 19:00:20.403693 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:20.403563    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv_odh-ai-gateway-infra(d28310ee-7c45-4a7f-bd1b-c6db37fd4a64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv" podUID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64"
Oct 30 19:00:20.407628 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.407598    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"
Oct 30 19:00:20.407749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.407624    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2" event={"ID":"7a647fbc-736b-4ea5-9375-fffb58047e03","Type":"ContainerDied","Data":"d5f95e0bc4f923694ddd5ced36d8fc3539a6001ec3736a3ef9a9a7c2bc3e0e05"}
Oct 30 19:00:20.422176 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.422119    6014 scope.go:122] "RemoveContainer" containerID="e9c208f1f9621c465d3ca99057f96c583290e639b7fcce975def96abe1db47c5"
Oct 30 19:00:20.479914 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.479877    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.480639 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.480610    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.481372 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.481345    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.482184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.482156    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.484494 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.484468    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.484601 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.484570    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q27x\" (UniqueName: \"kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-kube-api-access-7q27x\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.484662 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.484600    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.484662 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.484637    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.484784 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.484702    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.484784 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.484731    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.484893 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.484788    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.485295 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.485107    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.485577 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.485555    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.485770 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.485746    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.487787 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.487731    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.488776 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.488753    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.542463 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.542433    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.542612 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.542562    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q27x\" (UniqueName: \"kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-kube-api-access-7q27x\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.560436 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.560314    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"
Oct 30 19:00:20.576347 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.576315    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:20.630903 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.630826    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"]
Oct 30 19:00:20.637648 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.637614    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-gxfj2"]
Oct 30 19:00:20.668562 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.668525    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"]
Oct 30 19:00:20.687226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.687198    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-openshift-ca\") pod \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\" (UID: \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\") "
Oct 30 19:00:20.687395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.687245    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-kube-api-access-w7pt8\" (UniqueName: \"kubernetes.io/projected/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-kube-api-access-w7pt8\") pod \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\" (UID: \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\") "
Oct 30 19:00:20.687395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.687386    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-xks-ca\") pod \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\" (UID: \"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d\") "
Oct 30 19:00:20.688182 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.688111    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-openshift-ca" pod "c903929a-5bc5-426e-9bcc-caa2fe0e0d2d" (UID: "c903929a-5bc5-426e-9bcc-caa2fe0e0d2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:20.690879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.690847    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-kube-api-access-w7pt8" pod "c903929a-5bc5-426e-9bcc-caa2fe0e0d2d" (UID: "c903929a-5bc5-426e-9bcc-caa2fe0e0d2d"). InnerVolumeSpecName "kube-api-access-w7pt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:20.691004 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.690983    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-xks-ca" pod "c903929a-5bc5-426e-9bcc-caa2fe0e0d2d" (UID: "c903929a-5bc5-426e-9bcc-caa2fe0e0d2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:20.733125 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:20.733091    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898fcd26_1c1e_40e6_994d_619340f782cb.slice/crio-225d702175cf3078b001dd02c68c9a80207daf421e97b6c1041d02ffcb67449e WatchSource:0}: Error finding container 225d702175cf3078b001dd02c68c9a80207daf421e97b6c1041d02ffcb67449e: Status 404 returned error can't find the container with id 225d702175cf3078b001dd02c68c9a80207daf421e97b6c1041d02ffcb67449e
Oct 30 19:00:20.733607 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.733579    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"]
Oct 30 19:00:20.736012 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.735978    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:20.736116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.736057    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:20.736116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.736104    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:20.792360 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.792330    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.792599 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.792575    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7pt8\" (UniqueName: \"kubernetes.io/projected/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-kube-api-access-w7pt8\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.792739 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.792714    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.975791 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.975752    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"]
Oct 30 19:00:20.977621 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:20.977591    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62c6f22_f5b9_4061_9a75_ea765cf1a199.slice/crio-67d6f9b6c0d3ed6b6fd5ac08a3f96e2a69be612470a0db2d60e9469200aa1f15 WatchSource:0}: Error finding container 67d6f9b6c0d3ed6b6fd5ac08a3f96e2a69be612470a0db2d60e9469200aa1f15: Status 404 returned error can't find the container with id 67d6f9b6c0d3ed6b6fd5ac08a3f96e2a69be612470a0db2d60e9469200aa1f15
Oct 30 19:00:20.980443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.980409    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:20.980529 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.980475    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:20.980529 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:20.980506    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:21.417194 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.417160    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh" event={"ID":"a62c6f22-f5b9-4061-9a75-ea765cf1a199","Type":"ContainerStarted","Data":"c98c060f935cdfef02c3a8a4cfa642eeb5cd28f508bbd14b78d0b3945a0c2e9e"}
Oct 30 19:00:21.417676 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.417228    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh" event={"ID":"a62c6f22-f5b9-4061-9a75-ea765cf1a199","Type":"ContainerStarted","Data":"67d6f9b6c0d3ed6b6fd5ac08a3f96e2a69be612470a0db2d60e9469200aa1f15"}
Oct 30 19:00:21.419180 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.419129    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" event={"ID":"898fcd26-1c1e-40e6-994d-619340f782cb","Type":"ContainerStarted","Data":"2f0282cc95f3af422d3a83d09c30f8f687312f19747accc2a3a8b495bfe36093"}
Oct 30 19:00:21.419331 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.419199    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" event={"ID":"898fcd26-1c1e-40e6-994d-619340f782cb","Type":"ContainerStarted","Data":"225d702175cf3078b001dd02c68c9a80207daf421e97b6c1041d02ffcb67449e"}
Oct 30 19:00:21.423025 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.422964    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj" event={"ID":"c903929a-5bc5-426e-9bcc-caa2fe0e0d2d","Type":"ContainerDied","Data":"533efb72729e9c1622f4731eb6cae6240284a9dfa0f6e1169f60d86c1ad5ca3e"}
Oct 30 19:00:21.423025 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.422987    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"
Oct 30 19:00:21.423025 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.423006    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="533efb72729e9c1622f4731eb6cae6240284a9dfa0f6e1169f60d86c1ad5ca3e"
Oct 30 19:00:21.492946 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.492470    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f801899-0f12-4a1f-941d-de0d95106d93" path="/var/lib/kubelet/pods/5f801899-0f12-4a1f-941d-de0d95106d93/volumes"
Oct 30 19:00:21.493451 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.493429    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a647fbc-736b-4ea5-9375-fffb58047e03" path="/var/lib/kubelet/pods/7a647fbc-736b-4ea5-9375-fffb58047e03/volumes"
Oct 30 19:00:21.494323 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.494303    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9" path="/var/lib/kubelet/pods/bca4a6f1-c2b4-4285-b19e-3cbb46cf13a9/volumes"
Oct 30 19:00:21.541452 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.541386    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh" podStartSLOduration=1.541366603 podStartE2EDuration="1.541366603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:21.5370816 +0000 UTC m=+1260.180249214" watchObservedRunningTime="2026-10-30 19:00:21.541366603 +0000 UTC m=+1260.184534224"
Oct 30 19:00:21.579069 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.579032    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:21.820578 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.820484    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"
Oct 30 19:00:21.922655 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.922621    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-maas-api-service-ca\") pod \"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64\" (UID: \"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64\") "
Oct 30 19:00:21.922828 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.922707    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-kube-api-access-mv5br\" (UniqueName: \"kubernetes.io/projected/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-kube-api-access-mv5br\") pod \"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64\" (UID: \"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64\") "
Oct 30 19:00:21.923294 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.923234    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-maas-api-service-ca" pod "d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" (UID: "d28310ee-7c45-4a7f-bd1b-c6db37fd4a64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:21.924995 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:21.924968    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-kube-api-access-mv5br" pod "d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" (UID: "d28310ee-7c45-4a7f-bd1b-c6db37fd4a64"). InnerVolumeSpecName "kube-api-access-mv5br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:22.025609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.025580    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-maas-api-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:22.025951 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.025933    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv5br\" (UniqueName: \"kubernetes.io/projected/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64-kube-api-access-mv5br\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:22.121027 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.120958    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" podStartSLOduration=3.120940122 podStartE2EDuration="3.120940122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:22.002615887 +0000 UTC m=+1260.645783515" watchObservedRunningTime="2026-10-30 19:00:22.120940122 +0000 UTC m=+1260.764107737"
Oct 30 19:00:22.300221 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.299195    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:22.305314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.305276    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:00:22.305479 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.305339    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" podUID="898fcd26-1c1e-40e6-994d-619340f782cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 30 19:00:22.433328 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.433288    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv" event={"ID":"d28310ee-7c45-4a7f-bd1b-c6db37fd4a64","Type":"ContainerDied","Data":"05c3731822999382e9263cd5e1e4b73132f6160f5f3b5ccf167eeadb0a639cc7"}
Oct 30 19:00:22.433761 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.433308    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"
Oct 30 19:00:22.433761 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.433367    6014 scope.go:122] "RemoveContainer" containerID="08c4cc7ab8cdd744c8e520937b7441883de959081173d0cca298354060c3406e"
Oct 30 19:00:22.579169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.578934    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:00:22.579169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.578999    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh" podUID="a62c6f22-f5b9-4061-9a75-ea765cf1a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 19:00:22.615230 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.615192    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"]
Oct 30 19:00:22.617728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.617702    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889780-z29lv"]
Oct 30 19:00:22.780166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.779436    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"]
Oct 30 19:00:22.781173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.780521    6014 cpu_manager.go:414] "containerMap: removing container" podUID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" containerName="cleanup"
Oct 30 19:00:22.781173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.780547    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" containerName="cleanup"
Oct 30 19:00:22.781173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.780587    6014 cpu_manager.go:414] "containerMap: removing container" podUID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" containerName="cleanup"
Oct 30 19:00:22.781173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.780596    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" containerName="cleanup"
Oct 30 19:00:22.781173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.780615    6014 cpu_manager.go:414] "containerMap: removing container" podUID="c903929a-5bc5-426e-9bcc-caa2fe0e0d2d" containerName="revoke-keys"
Oct 30 19:00:22.781173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.780623    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c903929a-5bc5-426e-9bcc-caa2fe0e0d2d" containerName="revoke-keys"
Oct 30 19:00:22.781173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.780809    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" containerName="cleanup"
Oct 30 19:00:22.781173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.780826    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" containerName="cleanup"
Oct 30 19:00:22.781173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.780839    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" containerName="cleanup"
Oct 30 19:00:22.781173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.780851    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="c903929a-5bc5-426e-9bcc-caa2fe0e0d2d" containerName="revoke-keys"
Oct 30 19:00:22.807683 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.807653    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"
Oct 30 19:00:22.829007 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.828550    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"]
Oct 30 19:00:22.837496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.837421    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f86ed71-nvmrd\" (UID: \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"
Oct 30 19:00:22.837645 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.837593    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f86ed71-nvmrd\" (UID: \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"
Oct 30 19:00:22.837645 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.837643    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzlqt\" (UniqueName: \"kubernetes.io/projected/291abfe2-8f65-42e1-a425-957ac2a2c1d4-kube-api-access-fzlqt\") pod \"maas-api-revoke-sub-0f86ed71-nvmrd\" (UID: \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"
Oct 30 19:00:22.871166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.869231    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:00:22.871166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.869309    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:22.939590 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.939478    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f86ed71-nvmrd\" (UID: \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"
Oct 30 19:00:22.939756 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.939661    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzlqt\" (UniqueName: \"kubernetes.io/projected/291abfe2-8f65-42e1-a425-957ac2a2c1d4-kube-api-access-fzlqt\") pod \"maas-api-revoke-sub-0f86ed71-nvmrd\" (UID: \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"
Oct 30 19:00:22.940315 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.940064    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f86ed71-nvmrd\" (UID: \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"
Oct 30 19:00:22.940315 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.940129    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f86ed71-nvmrd\" (UID: \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"
Oct 30 19:00:22.944265 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:22.942773    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f86ed71-nvmrd\" (UID: \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"
Oct 30 19:00:23.009960 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:23.009927    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzlqt\" (UniqueName: \"kubernetes.io/projected/291abfe2-8f65-42e1-a425-957ac2a2c1d4-kube-api-access-fzlqt\") pod \"maas-api-revoke-sub-0f86ed71-nvmrd\" (UID: \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"
Oct 30 19:00:23.135267 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:23.135177    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"
Oct 30 19:00:23.293912 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:23.291947    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:00:23.293912 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:23.292010    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" podUID="898fcd26-1c1e-40e6-994d-619340f782cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 30 19:00:23.324209 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:23.323323    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:00:23.511258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:23.511177    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" path="/var/lib/kubelet/pods/d28310ee-7c45-4a7f-bd1b-c6db37fd4a64/volumes"
Oct 30 19:00:23.604868 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:23.604840    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:00:23.604960 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:23.604894    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh" podUID="a62c6f22-f5b9-4061-9a75-ea765cf1a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 19:00:23.605417 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:23.605376    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"]
Oct 30 19:00:24.292223 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:24.290921    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:00:24.292223 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:24.290986    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" podUID="898fcd26-1c1e-40e6-994d-619340f782cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 30 19:00:24.504190 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:24.502933    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd" event={"ID":"291abfe2-8f65-42e1-a425-957ac2a2c1d4","Type":"ContainerStarted","Data":"74d3431fe88bc5a73e12ba4513308aa4c2ee37f2ce3e51549cd40bbc4a5a5ef9"}
Oct 30 19:00:24.504190 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:24.503000    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd" event={"ID":"291abfe2-8f65-42e1-a425-957ac2a2c1d4","Type":"ContainerStarted","Data":"665638fe2128c2c359abe7cc81393a8a21818f417d60dd0bda8c47fac68e0cd2"}
Oct 30 19:00:24.597814 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:24.595244    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:00:24.597814 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:24.595304    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh" podUID="a62c6f22-f5b9-4061-9a75-ea765cf1a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 19:00:24.634916 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:24.634854    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd" podStartSLOduration=2.634837443 podStartE2EDuration="2.634837443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:24.633364504 +0000 UTC m=+1263.276532119" watchObservedRunningTime="2026-10-30 19:00:24.634837443 +0000 UTC m=+1263.278005058"
Oct 30 19:00:25.292647 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:25.292606    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:00:25.292964 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:25.292936    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" podUID="898fcd26-1c1e-40e6-994d-619340f782cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 30 19:00:25.512245 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:25.512214    6014 generic.go:356] "Generic (PLEG): container finished" podID="291abfe2-8f65-42e1-a425-957ac2a2c1d4" containerID="74d3431fe88bc5a73e12ba4513308aa4c2ee37f2ce3e51549cd40bbc4a5a5ef9" exitCode=0
Oct 30 19:00:25.512406 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:25.512298    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd" event={"ID":"291abfe2-8f65-42e1-a425-957ac2a2c1d4","Type":"ContainerDied","Data":"74d3431fe88bc5a73e12ba4513308aa4c2ee37f2ce3e51549cd40bbc4a5a5ef9"}
Oct 30 19:00:25.598171 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:25.580300    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:00:25.598171 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:25.580370    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh" podUID="a62c6f22-f5b9-4061-9a75-ea765cf1a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 19:00:26.313397 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:26.313286    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:00:26.313397 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:26.313390    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" podUID="898fcd26-1c1e-40e6-994d-619340f782cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 30 19:00:26.607720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:26.607421    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:00:26.607720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:26.607503    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh" podUID="a62c6f22-f5b9-4061-9a75-ea765cf1a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 19:00:26.839736 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:26.839713    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"
Oct 30 19:00:26.924896 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:26.924799    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-xks-ca\") pod \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\" (UID: \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\") "
Oct 30 19:00:26.924896 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:26.924877    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-openshift-ca\") pod \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\" (UID: \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\") "
Oct 30 19:00:26.925185 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:26.924923    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/291abfe2-8f65-42e1-a425-957ac2a2c1d4-kube-api-access-fzlqt\" (UniqueName: \"kubernetes.io/projected/291abfe2-8f65-42e1-a425-957ac2a2c1d4-kube-api-access-fzlqt\") pod \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\" (UID: \"291abfe2-8f65-42e1-a425-957ac2a2c1d4\") "
Oct 30 19:00:26.925283 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:26.925255    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-openshift-ca" pod "291abfe2-8f65-42e1-a425-957ac2a2c1d4" (UID: "291abfe2-8f65-42e1-a425-957ac2a2c1d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:26.925537 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:26.925519    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:26.927260 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:26.927236    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-xks-ca" pod "291abfe2-8f65-42e1-a425-957ac2a2c1d4" (UID: "291abfe2-8f65-42e1-a425-957ac2a2c1d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:26.927345 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:26.927310    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291abfe2-8f65-42e1-a425-957ac2a2c1d4-kube-api-access-fzlqt" pod "291abfe2-8f65-42e1-a425-957ac2a2c1d4" (UID: "291abfe2-8f65-42e1-a425-957ac2a2c1d4"). InnerVolumeSpecName "kube-api-access-fzlqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:27.027398 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:27.027364    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/291abfe2-8f65-42e1-a425-957ac2a2c1d4-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:27.027398 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:27.027400    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzlqt\" (UniqueName: \"kubernetes.io/projected/291abfe2-8f65-42e1-a425-957ac2a2c1d4-kube-api-access-fzlqt\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:27.290359 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:27.290269    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:00:27.290359 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:27.290336    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" podUID="898fcd26-1c1e-40e6-994d-619340f782cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 30 19:00:27.530912 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:27.530887    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"
Oct 30 19:00:27.531094 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:27.530879    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd" event={"ID":"291abfe2-8f65-42e1-a425-957ac2a2c1d4","Type":"ContainerDied","Data":"665638fe2128c2c359abe7cc81393a8a21818f417d60dd0bda8c47fac68e0cd2"}
Oct 30 19:00:27.531094 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:27.531024    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="665638fe2128c2c359abe7cc81393a8a21818f417d60dd0bda8c47fac68e0cd2"
Oct 30 19:00:27.577307 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:27.577271    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:00:27.577469 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:27.577337    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh" podUID="a62c6f22-f5b9-4061-9a75-ea765cf1a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 19:00:28.148955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.148901    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"]
Oct 30 19:00:28.149981 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.149955    6014 cpu_manager.go:414] "containerMap: removing container" podUID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" containerName="cleanup"
Oct 30 19:00:28.150093 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.149983    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28310ee-7c45-4a7f-bd1b-c6db37fd4a64" containerName="cleanup"
Oct 30 19:00:28.150093 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.150090    6014 cpu_manager.go:414] "containerMap: removing container" podUID="291abfe2-8f65-42e1-a425-957ac2a2c1d4" containerName="revoke-keys"
Oct 30 19:00:28.150246 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.150100    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291abfe2-8f65-42e1-a425-957ac2a2c1d4" containerName="revoke-keys"
Oct 30 19:00:28.150310 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.150279    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="291abfe2-8f65-42e1-a425-957ac2a2c1d4" containerName="revoke-keys"
Oct 30 19:00:28.154873 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.154850    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"
Oct 30 19:00:28.216195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.215738    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"]
Oct 30 19:00:28.216729 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.216523    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:00:28.216816 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.216780    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:28.256587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.254979    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d084ade5-bpx6s\" (UID: \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"
Oct 30 19:00:28.256587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.255053    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d084ade5-bpx6s\" (UID: \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"
Oct 30 19:00:28.256587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.255367    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6wtg\" (UniqueName: \"kubernetes.io/projected/51b1d03a-2932-4f68-8ef1-b35960cd7daf-kube-api-access-h6wtg\") pod \"maas-api-revoke-sub-d084ade5-bpx6s\" (UID: \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"
Oct 30 19:00:28.363591 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.358471    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d084ade5-bpx6s\" (UID: \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"
Oct 30 19:00:28.363591 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.358520    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d084ade5-bpx6s\" (UID: \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"
Oct 30 19:00:28.363591 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.358701    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6wtg\" (UniqueName: \"kubernetes.io/projected/51b1d03a-2932-4f68-8ef1-b35960cd7daf-kube-api-access-h6wtg\") pod \"maas-api-revoke-sub-d084ade5-bpx6s\" (UID: \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"
Oct 30 19:00:28.363591 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.361819    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d084ade5-bpx6s\" (UID: \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"
Oct 30 19:00:28.364501 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.364448    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d084ade5-bpx6s\" (UID: \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"
Oct 30 19:00:28.416613 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.416521    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6wtg\" (UniqueName: \"kubernetes.io/projected/51b1d03a-2932-4f68-8ef1-b35960cd7daf-kube-api-access-h6wtg\") pod \"maas-api-revoke-sub-d084ade5-bpx6s\" (UID: \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"
Oct 30 19:00:28.479775 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.479734    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"
Oct 30 19:00:28.504169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.503979    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:28.506426 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.505460    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:28.507043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.507010    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:00:28.702954 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.702443    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:28.703227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.703123    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:28.704344 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.704321    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:00:28.888235 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:28.888198    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"]
Oct 30 19:00:28.917393 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:28.917258    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1d03a_2932_4f68_8ef1_b35960cd7daf.slice/crio-46aa9454ac1bf7ab49792242878d77c9197e781d8d21d5c0acaa27fdcc85ab0e WatchSource:0}: Error finding container 46aa9454ac1bf7ab49792242878d77c9197e781d8d21d5c0acaa27fdcc85ab0e: Status 404 returned error can't find the container with id 46aa9454ac1bf7ab49792242878d77c9197e781d8d21d5c0acaa27fdcc85ab0e
Oct 30 19:00:29.551414 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:29.551316    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s" event={"ID":"51b1d03a-2932-4f68-8ef1-b35960cd7daf","Type":"ContainerStarted","Data":"5c5771ff6d1193c56cd5ccf857a767bd278a42d349ef9884c84dafb35c62b541"}
Oct 30 19:00:29.551414 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:29.551384    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s" event={"ID":"51b1d03a-2932-4f68-8ef1-b35960cd7daf","Type":"ContainerStarted","Data":"46aa9454ac1bf7ab49792242878d77c9197e781d8d21d5c0acaa27fdcc85ab0e"}
Oct 30 19:00:29.664160 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:29.664068    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s" podStartSLOduration=1.6640504489999999 podStartE2EDuration="1.664050449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:29.663387586 +0000 UTC m=+1268.306555201" watchObservedRunningTime="2026-10-30 19:00:29.664050449 +0000 UTC m=+1268.307218064"
Oct 30 19:00:30.560646 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:30.560557    6014 generic.go:356] "Generic (PLEG): container finished" podID="51b1d03a-2932-4f68-8ef1-b35960cd7daf" containerID="5c5771ff6d1193c56cd5ccf857a767bd278a42d349ef9884c84dafb35c62b541" exitCode=0
Oct 30 19:00:30.561038 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:30.560642    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s" event={"ID":"51b1d03a-2932-4f68-8ef1-b35960cd7daf","Type":"ContainerDied","Data":"5c5771ff6d1193c56cd5ccf857a767bd278a42d349ef9884c84dafb35c62b541"}
Oct 30 19:00:31.049606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:31.049570    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"]
Oct 30 19:00:31.053206 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:31.053180    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5450e8d4-krbf5"]
Oct 30 19:00:31.477857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:31.477828    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1024980a-606c-4968-a5f0-475a3cc63b92" path="/var/lib/kubelet/pods/1024980a-606c-4968-a5f0-475a3cc63b92/volumes"
Oct 30 19:00:31.731583 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:31.731515    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"
Oct 30 19:00:31.828434 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:31.828399    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b1d03a-2932-4f68-8ef1-b35960cd7daf-kube-api-access-h6wtg\" (UniqueName: \"kubernetes.io/projected/51b1d03a-2932-4f68-8ef1-b35960cd7daf-kube-api-access-h6wtg\") pod \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\" (UID: \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\") "
Oct 30 19:00:31.828627 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:31.828609    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-xks-ca\") pod \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\" (UID: \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\") "
Oct 30 19:00:31.828718 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:31.828658    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-openshift-ca\") pod \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\" (UID: \"51b1d03a-2932-4f68-8ef1-b35960cd7daf\") "
Oct 30 19:00:31.829132 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:31.829106    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-openshift-ca" pod "51b1d03a-2932-4f68-8ef1-b35960cd7daf" (UID: "51b1d03a-2932-4f68-8ef1-b35960cd7daf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:31.830834 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:31.830813    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-xks-ca" pod "51b1d03a-2932-4f68-8ef1-b35960cd7daf" (UID: "51b1d03a-2932-4f68-8ef1-b35960cd7daf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:31.830923 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:31.830900    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b1d03a-2932-4f68-8ef1-b35960cd7daf-kube-api-access-h6wtg" pod "51b1d03a-2932-4f68-8ef1-b35960cd7daf" (UID: "51b1d03a-2932-4f68-8ef1-b35960cd7daf"). InnerVolumeSpecName "kube-api-access-h6wtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:31.930809 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:31.930773    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:31.930809 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:31.930803    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6wtg\" (UniqueName: \"kubernetes.io/projected/51b1d03a-2932-4f68-8ef1-b35960cd7daf-kube-api-access-h6wtg\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:31.930809 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:31.930814    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b1d03a-2932-4f68-8ef1-b35960cd7daf-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:32.586357 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:32.586311    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s" event={"ID":"51b1d03a-2932-4f68-8ef1-b35960cd7daf","Type":"ContainerDied","Data":"46aa9454ac1bf7ab49792242878d77c9197e781d8d21d5c0acaa27fdcc85ab0e"}
Oct 30 19:00:32.586558 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:32.586395    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46aa9454ac1bf7ab49792242878d77c9197e781d8d21d5c0acaa27fdcc85ab0e"
Oct 30 19:00:32.586558 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:32.586335    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"
Oct 30 19:00:34.188271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.181205    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"]
Oct 30 19:00:34.188271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.182392    6014 cpu_manager.go:414] "containerMap: removing container" podUID="51b1d03a-2932-4f68-8ef1-b35960cd7daf" containerName="revoke-keys"
Oct 30 19:00:34.188271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.182417    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b1d03a-2932-4f68-8ef1-b35960cd7daf" containerName="revoke-keys"
Oct 30 19:00:34.188271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.182581    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b1d03a-2932-4f68-8ef1-b35960cd7daf" containerName="revoke-keys"
Oct 30 19:00:34.214166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.207445    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.463273 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.461805    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.588248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.586696    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.588248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.586784    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.588248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.586859    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.588248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.586896    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.588248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.587050    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5278d58c-cb56-4847-88d1-e366fad02e4a-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.588248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.587118    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.588248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.587183    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.588248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.587217    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.588248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.587319    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqjdg\" (UniqueName: \"kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-kube-api-access-vqjdg\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.588248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.587687    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.650499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.650466    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"]
Oct 30 19:00:34.728234 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.711736    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5278d58c-cb56-4847-88d1-e366fad02e4a-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.739750 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.739709    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.749446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.749406    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.753164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.749659    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.757428 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.753452    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqjdg\" (UniqueName: \"kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-kube-api-access-vqjdg\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.781166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.757750    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.781502 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.781473    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.781731 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.781708    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.781858 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.781843    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.782168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.759384    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.782302 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.744376    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5278d58c-cb56-4847-88d1-e366fad02e4a-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.782408 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.758504    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.782498 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.759052    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.782692 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.782664    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.792166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.787539    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.845176 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.840473    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.870203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.870104    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqjdg\" (UniqueName: \"kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-kube-api-access-vqjdg\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.895166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.883263    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.899163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.896772    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:34.906173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:34.902666    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:35.039223 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:35.028991    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:35.064165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:35.063616    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:35.162166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:35.161634    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:35.205739 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:35.204403    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:35.291261 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:35.264810    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded"
Oct 30 19:00:35.510203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:35.500104    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:35.682169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:35.679014    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:00:35.733168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:35.709228    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:00:35.804161 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:35.789490    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:36.273444 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:35.920240    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:36.319848 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.278559    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:00:36.319848 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.303278    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:00:36.319848 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.105614    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:00:36.319848 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.105647    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:00:36.319848 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.225088    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:36.319848 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.305321    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:00:36.319848 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.305373    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:00:36.319848 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.305412    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:00:36.413233 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.404591    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"]
Oct 30 19:00:36.480771 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.480740    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"
Oct 30 19:00:36.506295 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.506251    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:36.618113 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.540501    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:36.621120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.543326    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:00:36.715361 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.715263    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:36.737568 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.737539    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"]
Oct 30 19:00:36.762314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.762282    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:00:36.796651 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.796604    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:00:36.835482 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.835338    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:00:36.835819 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.835799    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:00:36.844063 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.844027    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9403a0ab-9nmlc\" (UID: \"272737d7-b2ef-4706-8cb7-bf089905b287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"
Oct 30 19:00:36.848671 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.848638    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9403a0ab-9nmlc\" (UID: \"272737d7-b2ef-4706-8cb7-bf089905b287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"
Oct 30 19:00:36.851630 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.844849    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:00:36.855657 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.855633    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:00:36.868531 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.868406    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwnwk\" (UniqueName: \"kubernetes.io/projected/272737d7-b2ef-4706-8cb7-bf089905b287-kube-api-access-qwnwk\") pod \"maas-api-revoke-sub-9403a0ab-9nmlc\" (UID: \"272737d7-b2ef-4706-8cb7-bf089905b287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"
Oct 30 19:00:36.970586 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.970494    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwnwk\" (UniqueName: \"kubernetes.io/projected/272737d7-b2ef-4706-8cb7-bf089905b287-kube-api-access-qwnwk\") pod \"maas-api-revoke-sub-9403a0ab-9nmlc\" (UID: \"272737d7-b2ef-4706-8cb7-bf089905b287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"
Oct 30 19:00:36.970586 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.970579    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9403a0ab-9nmlc\" (UID: \"272737d7-b2ef-4706-8cb7-bf089905b287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"
Oct 30 19:00:36.970833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.970812    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9403a0ab-9nmlc\" (UID: \"272737d7-b2ef-4706-8cb7-bf089905b287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"
Oct 30 19:00:36.971500 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.971461    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9403a0ab-9nmlc\" (UID: \"272737d7-b2ef-4706-8cb7-bf089905b287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"
Oct 30 19:00:36.973447 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.973418    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9403a0ab-9nmlc\" (UID: \"272737d7-b2ef-4706-8cb7-bf089905b287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"
Oct 30 19:00:36.991401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.991369    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"]
Oct 30 19:00:36.994354 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:36.994320    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5278d58c_cb56_4847_88d1_e366fad02e4a.slice/crio-4eed332239540d1448478845d4916ddc2dd9f50a6c5261fc46e72065fbf2db13 WatchSource:0}: Error finding container 4eed332239540d1448478845d4916ddc2dd9f50a6c5261fc46e72065fbf2db13: Status 404 returned error can't find the container with id 4eed332239540d1448478845d4916ddc2dd9f50a6c5261fc46e72065fbf2db13
Oct 30 19:00:36.998273 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.998093    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:36.998273 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.998214    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:36.998273 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:36.998266    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:37.028437 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:37.028409    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwnwk\" (UniqueName: \"kubernetes.io/projected/272737d7-b2ef-4706-8cb7-bf089905b287-kube-api-access-qwnwk\") pod \"maas-api-revoke-sub-9403a0ab-9nmlc\" (UID: \"272737d7-b2ef-4706-8cb7-bf089905b287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"
Oct 30 19:00:37.276166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:37.276048    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"
Oct 30 19:00:37.678198 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:37.678085    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"]
Oct 30 19:00:37.700395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:37.700366    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm" event={"ID":"5278d58c-cb56-4847-88d1-e366fad02e4a","Type":"ContainerStarted","Data":"c4a4a5b55625e737dd50848daccb238c5a1a36ceabdd35a3e3a5ad39b8d511d8"}
Oct 30 19:00:37.700597 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:37.700425    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm" event={"ID":"5278d58c-cb56-4847-88d1-e366fad02e4a","Type":"ContainerStarted","Data":"4eed332239540d1448478845d4916ddc2dd9f50a6c5261fc46e72065fbf2db13"}
Oct 30 19:00:37.993374 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:37.993265    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm" podStartSLOduration=3.993249105 podStartE2EDuration="3.993249105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:37.990771104 +0000 UTC m=+1276.633938733" watchObservedRunningTime="2026-10-30 19:00:37.993249105 +0000 UTC m=+1276.636416788"
Oct 30 19:00:38.176163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:38.175487    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:38.185906 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:38.185857    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:00:38.186105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:38.185937    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm" podUID="5278d58c-cb56-4847-88d1-e366fad02e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 19:00:38.709842 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:38.709803    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc" event={"ID":"272737d7-b2ef-4706-8cb7-bf089905b287","Type":"ContainerStarted","Data":"1144f5c9b248cadb44eb5d7bb884353b275b4a74c46e114dba0f849c8a509766"}
Oct 30 19:00:38.710370 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:38.709869    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc" event={"ID":"272737d7-b2ef-4706-8cb7-bf089905b287","Type":"ContainerStarted","Data":"b38b53e2c7a1f06205ff73da5b442c177182d2f3ba19cb9b0527e9183b4af902"}
Oct 30 19:00:38.835869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:38.835636    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc" podStartSLOduration=2.835615675 podStartE2EDuration="2.835615675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:38.830062305 +0000 UTC m=+1277.473229928" watchObservedRunningTime="2026-10-30 19:00:38.835615675 +0000 UTC m=+1277.478783291"
Oct 30 19:00:39.171166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:39.167576    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:00:39.171166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:39.167646    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm" podUID="5278d58c-cb56-4847-88d1-e366fad02e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 19:00:39.719081 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:39.719043    6014 generic.go:356] "Generic (PLEG): container finished" podID="272737d7-b2ef-4706-8cb7-bf089905b287" containerID="1144f5c9b248cadb44eb5d7bb884353b275b4a74c46e114dba0f849c8a509766" exitCode=0
Oct 30 19:00:39.719513 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:39.719116    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc" event={"ID":"272737d7-b2ef-4706-8cb7-bf089905b287","Type":"ContainerDied","Data":"1144f5c9b248cadb44eb5d7bb884353b275b4a74c46e114dba0f849c8a509766"}
Oct 30 19:00:40.172204 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:40.167742    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:00:40.172204 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:40.167836    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm" podUID="5278d58c-cb56-4847-88d1-e366fad02e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 19:00:40.928657 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:40.928633    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"
Oct 30 19:00:41.037708 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.037635    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-xks-ca\") pod \"272737d7-b2ef-4706-8cb7-bf089905b287\" (UID: \"272737d7-b2ef-4706-8cb7-bf089905b287\") "
Oct 30 19:00:41.037708 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.037698    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-openshift-ca\") pod \"272737d7-b2ef-4706-8cb7-bf089905b287\" (UID: \"272737d7-b2ef-4706-8cb7-bf089905b287\") "
Oct 30 19:00:41.037954 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.037938    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/272737d7-b2ef-4706-8cb7-bf089905b287-kube-api-access-qwnwk\" (UniqueName: \"kubernetes.io/projected/272737d7-b2ef-4706-8cb7-bf089905b287-kube-api-access-qwnwk\") pod \"272737d7-b2ef-4706-8cb7-bf089905b287\" (UID: \"272737d7-b2ef-4706-8cb7-bf089905b287\") "
Oct 30 19:00:41.038066 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.038045    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-openshift-ca" pod "272737d7-b2ef-4706-8cb7-bf089905b287" (UID: "272737d7-b2ef-4706-8cb7-bf089905b287"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:41.038517 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.038495    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:41.040307 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.040285    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-xks-ca" pod "272737d7-b2ef-4706-8cb7-bf089905b287" (UID: "272737d7-b2ef-4706-8cb7-bf089905b287"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:41.040406 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.040327    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272737d7-b2ef-4706-8cb7-bf089905b287-kube-api-access-qwnwk" pod "272737d7-b2ef-4706-8cb7-bf089905b287" (UID: "272737d7-b2ef-4706-8cb7-bf089905b287"). InnerVolumeSpecName "kube-api-access-qwnwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:41.140024 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.139980    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/272737d7-b2ef-4706-8cb7-bf089905b287-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:41.140024 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.140017    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwnwk\" (UniqueName: \"kubernetes.io/projected/272737d7-b2ef-4706-8cb7-bf089905b287-kube-api-access-qwnwk\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:41.172168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.171617    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:00:41.172168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.171684    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm" podUID="5278d58c-cb56-4847-88d1-e366fad02e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 19:00:41.737275 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.737177    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"
Oct 30 19:00:41.737275 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.737196    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc" event={"ID":"272737d7-b2ef-4706-8cb7-bf089905b287","Type":"ContainerDied","Data":"b38b53e2c7a1f06205ff73da5b442c177182d2f3ba19cb9b0527e9183b4af902"}
Oct 30 19:00:41.737275 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:41.737260    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b38b53e2c7a1f06205ff73da5b442c177182d2f3ba19cb9b0527e9183b4af902"
Oct 30 19:00:42.172170 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:42.169544    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:00:42.172170 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:42.169623    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm" podUID="5278d58c-cb56-4847-88d1-e366fad02e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 19:00:43.065120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:43.065083    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"]
Oct 30 19:00:43.099375 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:43.099342    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a880054-tns4d"]
Oct 30 19:00:43.165468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:43.165433    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:00:43.165636 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:43.165488    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm" podUID="5278d58c-cb56-4847-88d1-e366fad02e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 19:00:43.477204 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:43.477100    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed200bf7-d482-4ec9-b125-84c9eeea2c4e" path="/var/lib/kubelet/pods/ed200bf7-d482-4ec9-b125-84c9eeea2c4e/volumes"
Oct 30 19:00:44.394489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:44.394448    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:44.394933 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:44.394908    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:44.396186 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:44.396162    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:00:49.257255 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.257216    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"]
Oct 30 19:00:49.258106 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.258085    6014 cpu_manager.go:414] "containerMap: removing container" podUID="272737d7-b2ef-4706-8cb7-bf089905b287" containerName="revoke-keys"
Oct 30 19:00:49.258233 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.258108    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272737d7-b2ef-4706-8cb7-bf089905b287" containerName="revoke-keys"
Oct 30 19:00:49.258374 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.258359    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="272737d7-b2ef-4706-8cb7-bf089905b287" containerName="revoke-keys"
Oct 30 19:00:49.263461 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.263441    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"
Oct 30 19:00:49.284589 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.284563    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66b4ba65-ldqz2\" (UID: \"3f582f52-06a8-4f9e-806d-06bf591b64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"
Oct 30 19:00:49.284742 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.284721    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66b4ba65-ldqz2\" (UID: \"3f582f52-06a8-4f9e-806d-06bf591b64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"
Oct 30 19:00:49.284787 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.284766    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x69kp\" (UniqueName: \"kubernetes.io/projected/3f582f52-06a8-4f9e-806d-06bf591b64e6-kube-api-access-x69kp\") pod \"maas-api-revoke-sub-66b4ba65-ldqz2\" (UID: \"3f582f52-06a8-4f9e-806d-06bf591b64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"
Oct 30 19:00:49.306895 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.306867    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"]
Oct 30 19:00:49.311167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.311123    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.317478 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.317457    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"]
Oct 30 19:00:49.347182 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.339266    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:49.347182 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.339463    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:00:49.372879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.372848    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"]
Oct 30 19:00:49.385916 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.385879    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42816184-e4b9-4216-8298-60ec4aebe37f-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.386112 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.385947    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42816184-e4b9-4216-8298-60ec4aebe37f-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.386112 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.386062    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66b4ba65-ldqz2\" (UID: \"3f582f52-06a8-4f9e-806d-06bf591b64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"
Oct 30 19:00:49.386271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.386117    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.386271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.386169    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x69kp\" (UniqueName: \"kubernetes.io/projected/3f582f52-06a8-4f9e-806d-06bf591b64e6-kube-api-access-x69kp\") pod \"maas-api-revoke-sub-66b4ba65-ldqz2\" (UID: \"3f582f52-06a8-4f9e-806d-06bf591b64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"
Oct 30 19:00:49.386271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.386260    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.386439 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.386297    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.386439 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.386360    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.386439 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.386390    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.386439 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.386427    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.386645 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.386483    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxdk7\" (UniqueName: \"kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-kube-api-access-fxdk7\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.386645 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.386589    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66b4ba65-ldqz2\" (UID: \"3f582f52-06a8-4f9e-806d-06bf591b64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"
Oct 30 19:00:49.386771 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.386746    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66b4ba65-ldqz2\" (UID: \"3f582f52-06a8-4f9e-806d-06bf591b64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"
Oct 30 19:00:49.389548 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.389521    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66b4ba65-ldqz2\" (UID: \"3f582f52-06a8-4f9e-806d-06bf591b64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"
Oct 30 19:00:49.443749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.443722    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x69kp\" (UniqueName: \"kubernetes.io/projected/3f582f52-06a8-4f9e-806d-06bf591b64e6-kube-api-access-x69kp\") pod \"maas-api-revoke-sub-66b4ba65-ldqz2\" (UID: \"3f582f52-06a8-4f9e-806d-06bf591b64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"
Oct 30 19:00:49.487641 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.487610    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.487641 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.487643    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.487884 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.487689    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.487884 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.487806    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.487884 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.487846    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.488042 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.487902    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxdk7\" (UniqueName: \"kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-kube-api-access-fxdk7\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.488042 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.488028    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.488155 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.488078    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.488214 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.488191    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42816184-e4b9-4216-8298-60ec4aebe37f-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.488285 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.488260    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42816184-e4b9-4216-8298-60ec4aebe37f-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.488285 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.488268    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.488424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.488396    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.488712 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.488668    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.489113 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.489085    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42816184-e4b9-4216-8298-60ec4aebe37f-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.490530 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.490507    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.491066 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.491044    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42816184-e4b9-4216-8298-60ec4aebe37f-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.544972 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.544894    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.545120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.545046    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxdk7\" (UniqueName: \"kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-kube-api-access-fxdk7\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.578290 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.578260    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"
Oct 30 19:00:49.625888 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.625854    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:49.961776 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.961750    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"]
Oct 30 19:00:49.961982 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:49.961960    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f582f52_06a8_4f9e_806d_06bf591b64e6.slice/crio-e695875b53d2e268f217a091fed9bfddef0949e8b69f0802be0201bdff057dd0 WatchSource:0}: Error finding container e695875b53d2e268f217a091fed9bfddef0949e8b69f0802be0201bdff057dd0: Status 404 returned error can't find the container with id e695875b53d2e268f217a091fed9bfddef0949e8b69f0802be0201bdff057dd0
Oct 30 19:00:49.970059 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.970034    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"]
Oct 30 19:00:49.975598 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.975582    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:49.995016 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.994985    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:49.995133 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.995044    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:49.995205 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.995131    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:49.995275 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.995258    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:49.995315 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.995286    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c565e581-0906-402f-8dbe-16154942ba9e-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:49.995360 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.995325    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:49.995360 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.995341    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd6qt\" (UniqueName: \"kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-kube-api-access-qd6qt\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:49.995481 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.995457    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c565e581-0906-402f-8dbe-16154942ba9e-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:49.995521 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:49.995500    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.034508 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.034471    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"]
Oct 30 19:00:50.079904 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:00:50.079801    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42816184_e4b9_4216_8298_60ec4aebe37f.slice/crio-c3f31d0031abf2d633765af024927e663662d850cd50ebb9aca74326f65076c6 WatchSource:0}: Error finding container c3f31d0031abf2d633765af024927e663662d850cd50ebb9aca74326f65076c6: Status 404 returned error can't find the container with id c3f31d0031abf2d633765af024927e663662d850cd50ebb9aca74326f65076c6
Oct 30 19:00:50.082954 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.082916    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:50.083092 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.083006    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:50.083092 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.083053    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:50.084847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.084825    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"]
Oct 30 19:00:50.097298 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.097263    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.097466 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.097344    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.097466 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.097388    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.097605 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.097486    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.097605 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.097516    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c565e581-0906-402f-8dbe-16154942ba9e-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.097605 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.097574    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.097605 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.097599    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd6qt\" (UniqueName: \"kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-kube-api-access-qd6qt\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.097802 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.097658    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c565e581-0906-402f-8dbe-16154942ba9e-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.097802 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.097674    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.097802 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.097713    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.097802 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.097722    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.098296 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.098074    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.098436 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.098390    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.098720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.098684    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c565e581-0906-402f-8dbe-16154942ba9e-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.101639 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.101616    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.101806 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.101786    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c565e581-0906-402f-8dbe-16154942ba9e-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.154122 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.154097    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd6qt\" (UniqueName: \"kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-kube-api-access-qd6qt\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.154308 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.154245    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.159513 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.159487    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:00:50.234574 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.234493    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"]
Oct 30 19:00:50.243175 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.243117    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.298705 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.298667    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"]
Oct 30 19:00:50.303533 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.303506    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.303724 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.303668    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.303724 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.303710    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.303826 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.303764    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.303898 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.303827    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.303898 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.303888    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b2dba2-c676-4658-8dc3-d47d9782b180-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.304325 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.304173    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.304325 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.304211    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9dqh\" (UniqueName: \"kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-kube-api-access-q9dqh\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.304325 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.304314    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.309891 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.309863    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:50.406282 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.406242    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.406439 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.406304    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.406439 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.406424    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.406564 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.406467    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.406564 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.406498    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.406564 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.406550    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.406725 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.406595    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b2dba2-c676-4658-8dc3-d47d9782b180-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.406802 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.406780    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.406865 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.406832    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9dqh\" (UniqueName: \"kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-kube-api-access-q9dqh\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.407757 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.407726    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.408418 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.408390    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.408668 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.408644    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.409338 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.409313    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b2dba2-c676-4658-8dc3-d47d9782b180-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.410206 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.410183    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.412664 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.412635    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.414290 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.414264    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.464192 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.464114    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9dqh\" (UniqueName: \"kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-kube-api-access-q9dqh\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.465899 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.465872    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.523354 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.523264    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"]
Oct 30 19:00:50.528938 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.528916    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.564865 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.564830    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:50.588992 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.588953    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"]
Oct 30 19:00:50.611058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.611021    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.611245 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.611126    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.611318 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.611279    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.611366 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.611339    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.611488 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.611462    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.611546 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.611531    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzbc\" (UniqueName: \"kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-kube-api-access-6kzbc\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.611697 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.611681    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.611782 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.611710    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.611782 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.611750    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.713424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.713367    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.713590 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.713436    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.713590 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.713475    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.713590 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.713545    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.713590 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.713581    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.713838 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.713662    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.713838 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.713730    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.713939 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.713860    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.713939 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.713917    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzbc\" (UniqueName: \"kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-kube-api-access-6kzbc\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.715056 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.715029    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.715313 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.715294    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.718781 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.718616    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.718781 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.718721    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.719012 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.718968    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.719678 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.719221    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.719678 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.719597    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.722196 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.721443    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"]
Oct 30 19:00:50.725863 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.725834    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:50.725929 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.725897    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:50.725967 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.725929    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:50.784382 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.784284    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzbc\" (UniqueName: \"kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-kube-api-access-6kzbc\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.785052 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.785031    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.811495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.811453    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2" event={"ID":"3f582f52-06a8-4f9e-806d-06bf591b64e6","Type":"ContainerStarted","Data":"2933bed322a852a5c33d91f6987e78f5ce353c027df8675c6295a98b2cb49956"}
Oct 30 19:00:50.811669 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.811541    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2" event={"ID":"3f582f52-06a8-4f9e-806d-06bf591b64e6","Type":"ContainerStarted","Data":"e695875b53d2e268f217a091fed9bfddef0949e8b69f0802be0201bdff057dd0"}
Oct 30 19:00:50.814291 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.814180    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" event={"ID":"42816184-e4b9-4216-8298-60ec4aebe37f","Type":"ContainerStarted","Data":"43353d2ed2af088db455094553290b8b0de9e4bcc5523c521bbea0293e61b756"}
Oct 30 19:00:50.814291 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.814237    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" event={"ID":"42816184-e4b9-4216-8298-60ec4aebe37f","Type":"ContainerStarted","Data":"c3f31d0031abf2d633765af024927e663662d850cd50ebb9aca74326f65076c6"}
Oct 30 19:00:50.815891 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.815865    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" event={"ID":"c565e581-0906-402f-8dbe-16154942ba9e","Type":"ContainerStarted","Data":"882bcbe54bdaf7100f08e2576499c77a6b11d896633113d361d4f5ff635c2f3b"}
Oct 30 19:00:50.844712 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.844487    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:50.986163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:50.984519    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2" podStartSLOduration=1.984495827 podStartE2EDuration="1.984495827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:50.945637307 +0000 UTC m=+1289.588804922" watchObservedRunningTime="2026-10-30 19:00:50.984495827 +0000 UTC m=+1289.627663441"
Oct 30 19:00:51.063917 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:51.059559    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"]
Oct 30 19:00:51.115385 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:51.098422    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podStartSLOduration=2.098401401 podStartE2EDuration="2.098401401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:51.066739693 +0000 UTC m=+1289.709907308" watchObservedRunningTime="2026-10-30 19:00:51.098401401 +0000 UTC m=+1289.741569016"
Oct 30 19:00:51.157434 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:51.157305    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:51.164427 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:51.157540    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:51.164427 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:51.157623    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:51.665948 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:51.664458    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:00:51.665948 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:51.664518    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:00:52.023504 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.023462    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:00:52.023855 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.023833    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"]
Oct 30 19:00:52.127266 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.113935    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:52.178313 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.178254    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:52.203167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.178578    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:52.204039 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.186448    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2" event={"ID":"3f582f52-06a8-4f9e-806d-06bf591b64e6","Type":"ContainerDied","Data":"2933bed322a852a5c33d91f6987e78f5ce353c027df8675c6295a98b2cb49956"}
Oct 30 19:00:52.204280 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.186018    6014 generic.go:356] "Generic (PLEG): container finished" podID="3f582f52-06a8-4f9e-806d-06bf591b64e6" containerID="2933bed322a852a5c33d91f6987e78f5ce353c027df8675c6295a98b2cb49956" exitCode=0
Oct 30 19:00:52.285182 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.282568    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" event={"ID":"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2","Type":"ContainerStarted","Data":"36dbf4d17181684caa9cac3f9189278503720b2e744985f54f15779dbce3883c"}
Oct 30 19:00:52.419702 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.419602    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" event={"ID":"58b2dba2-c676-4658-8dc3-d47d9782b180","Type":"ContainerStarted","Data":"e8a783c5a7e83eabca1393180399c7c8509e32d0eefc72d48ea1049f40633096"}
Oct 30 19:00:52.530168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.526747    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" event={"ID":"c565e581-0906-402f-8dbe-16154942ba9e","Type":"ContainerStarted","Data":"758c22e6d437c2ab93c918557e173533f3862b78d65b86536582211825979420"}
Oct 30 19:00:52.677702 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.653522    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:00:52.677702 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.653581    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:00:52.696342 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.696291    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:00:52.813657 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.813247    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:52.822323 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.822283    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:52.841614 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.841564    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:52.861456 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.849683    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" podStartSLOduration=3.84966311 podStartE2EDuration="3.84966311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:52.779819796 +0000 UTC m=+1291.422987411" watchObservedRunningTime="2026-10-30 19:00:52.84966311 +0000 UTC m=+1291.492830726"
Oct 30 19:00:52.889296 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:52.889241    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:53.289400 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.289272    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:53.312342 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.312307    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:00:53.312669 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.312646    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:00:53.312873 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.312847    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:00:53.690168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.653763    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:00:53.724166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.697401    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:00:53.724166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.700439    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:53.824327 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.824269    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:00:53.853535 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:00:53.853497    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:00:53.877184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.862890    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:53.877184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.863344    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:00:53.877184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.866466    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:53.890181 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.888244    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:00:53.891363 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.891226    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:00:54.045231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.916777    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:00:54.045231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.944377    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 19:00:54.045231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.944414    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" event={"ID":"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2","Type":"ContainerStarted","Data":"686b6a276fd33a82fd58defbd76458c55c1cbbfd8a0b12d55ac2d82f3aabbd2e"}
Oct 30 19:00:54.045231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.944452    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" event={"ID":"58b2dba2-c676-4658-8dc3-d47d9782b180","Type":"ContainerStarted","Data":"3a67e298f1ea43f5a7100542fcbfef257a2602621e1c1bea56569811822761a5"}
Oct 30 19:00:54.045231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.944509    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:00:54.045231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.944551    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:00:54.045231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.944588    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:00:54.045231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:53.944627    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:00:54.450714 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.450681    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:00:54.450981 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.450955    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:00:54.451416 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.451362    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" podStartSLOduration=4.451343469 podStartE2EDuration="4.451343469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:54.324431438 +0000 UTC m=+1292.967599052" watchObservedRunningTime="2026-10-30 19:00:54.451343469 +0000 UTC m=+1293.094511084"
Oct 30 19:00:54.575522 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.575461    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:54.673164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.672166    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:00:54.673164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.672247    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:00:54.698167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.674408    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:00:54.750252 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.708572    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:00:54.750252 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.708655    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 19:00:54.750252 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.710226    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:54.850189 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.829980    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:54.871453 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.871408    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:54.880313 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.880263    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:00:54.902768 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.902728    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:00:54.926184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.903435    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:00:54.927570 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.927531    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:00:55.034450 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.012244    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:00:55.034690 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:54.929075    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:00:55.100204 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.096198    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:00:55.100204 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.096262    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 19:00:55.423263 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.413465    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:00:55.423263 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.413525    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:00:55.452163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.449067    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" podStartSLOduration=5.449042085 podStartE2EDuration="5.449042085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:55.322996233 +0000 UTC m=+1293.966163889" watchObservedRunningTime="2026-10-30 19:00:55.449042085 +0000 UTC m=+1294.092209702"
Oct 30 19:00:55.598173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.595076    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:00:55.598173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.595219    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 19:00:55.664429 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.656219    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:00:55.664429 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.656281    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:00:55.664429 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.656354    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded"
Oct 30 19:00:55.801709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.778046    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:55.847815 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.835235    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:00:55.847815 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.847334    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:00:55.847815 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.847390    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 19:00:55.955968 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.954080    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:55.955968 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.954407    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:00:55.955968 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:55.954677    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:00:56.041606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.041560    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:00:56.135164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.097811    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:00:56.474429 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.474303    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:00:56.570604 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.570439    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:00:56.580531 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.580491    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:00:56.580849 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.580815    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 19:00:56.697411 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.697371    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:00:56.698521 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.698486    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:00:56.698714 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.697807    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"
Oct 30 19:00:56.783229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.753792    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:56.866747 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.801281    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:56.930175 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.928230    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:00:56.930175 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.928293    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 19:00:56.954194 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.949039    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f582f52-06a8-4f9e-806d-06bf591b64e6-kube-api-access-x69kp\" (UniqueName: \"kubernetes.io/projected/3f582f52-06a8-4f9e-806d-06bf591b64e6-kube-api-access-x69kp\") pod \"3f582f52-06a8-4f9e-806d-06bf591b64e6\" (UID: \"3f582f52-06a8-4f9e-806d-06bf591b64e6\") "
Oct 30 19:00:56.954194 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.949130    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-openshift-ca\") pod \"3f582f52-06a8-4f9e-806d-06bf591b64e6\" (UID: \"3f582f52-06a8-4f9e-806d-06bf591b64e6\") "
Oct 30 19:00:56.954194 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.949497    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-xks-ca\") pod \"3f582f52-06a8-4f9e-806d-06bf591b64e6\" (UID: \"3f582f52-06a8-4f9e-806d-06bf591b64e6\") "
Oct 30 19:00:56.970172 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.966647    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:56.992483 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.982756    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:56.992483 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:56.983360    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:57.040215 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:57.034791    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-openshift-ca" pod "3f582f52-06a8-4f9e-806d-06bf591b64e6" (UID: "3f582f52-06a8-4f9e-806d-06bf591b64e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:57.079327 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:57.079278    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:57.120578 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:57.106807    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:00:57.120578 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:57.109844    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:57.175167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:57.159960    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:57.647281 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:57.647243    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-xks-ca" pod "3f582f52-06a8-4f9e-806d-06bf591b64e6" (UID: "3f582f52-06a8-4f9e-806d-06bf591b64e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:57.982046 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:57.647257    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:00:58.146320 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.085583    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 19:00:58.378172 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.028701    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:58.378172 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:57.994465    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"
Oct 30 19:00:58.378172 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:57.654384    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:00:58.378172 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.376296    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:00:58.378172 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.105264    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.403177    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded"
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:57.679742    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.407494    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:57.994599    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f582f52-06a8-4f9e-806d-06bf591b64e6-kube-api-access-x69kp" pod "3f582f52-06a8-4f9e-806d-06bf591b64e6" (UID: "3f582f52-06a8-4f9e-806d-06bf591b64e6"). InnerVolumeSpecName "kube-api-access-x69kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.076723    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.407617    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.114295    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.114341    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.211388    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.429446    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.429502    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.116435    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.431725    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.458850    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f582f52-06a8-4f9e-806d-06bf591b64e6-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.458940    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x69kp\" (UniqueName: \"kubernetes.io/projected/3f582f52-06a8-4f9e-806d-06bf591b64e6-kube-api-access-x69kp\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.460094    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.460240    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2" event={"ID":"3f582f52-06a8-4f9e-806d-06bf591b64e6","Type":"ContainerDied","Data":"e695875b53d2e268f217a091fed9bfddef0949e8b69f0802be0201bdff057dd0"}
Oct 30 19:00:58.525166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.467266    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e695875b53d2e268f217a091fed9bfddef0949e8b69f0802be0201bdff057dd0"
Oct 30 19:00:58.671317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.651544    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:00:58.671317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.651624    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:00:58.671317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.651945    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"]
Oct 30 19:00:58.671317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.653820    6014 cpu_manager.go:414] "containerMap: removing container" podUID="3f582f52-06a8-4f9e-806d-06bf591b64e6" containerName="revoke-keys"
Oct 30 19:00:58.671317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.653842    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f582f52-06a8-4f9e-806d-06bf591b64e6" containerName="revoke-keys"
Oct 30 19:00:58.702165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.681804    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f582f52-06a8-4f9e-806d-06bf591b64e6" containerName="revoke-keys"
Oct 30 19:00:58.707856 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.705761    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:00:58.707856 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.706648    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 19:00:58.899173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.895073    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"
Oct 30 19:00:58.923535 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.923327    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:00:58.939518 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.937314    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:00:58.939518 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.937399    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 19:00:58.979606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:58.979574    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:00:59.002411 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.002372    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b839a2d9-58p2k\" (UID: \"ccae37a5-4937-41df-9cae-6f0a76a40b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"
Oct 30 19:00:59.003274 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.003206    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqbgp\" (UniqueName: \"kubernetes.io/projected/ccae37a5-4937-41df-9cae-6f0a76a40b38-kube-api-access-vqbgp\") pod \"maas-api-revoke-sub-b839a2d9-58p2k\" (UID: \"ccae37a5-4937-41df-9cae-6f0a76a40b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"
Oct 30 19:00:59.020010 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.019977    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b839a2d9-58p2k\" (UID: \"ccae37a5-4937-41df-9cae-6f0a76a40b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"
Oct 30 19:00:59.057162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.054617    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Oct 30 19:00:59.080897 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.080854    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"]
Oct 30 19:00:59.107177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.094444    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:59.203217 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.186131    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b839a2d9-58p2k\" (UID: \"ccae37a5-4937-41df-9cae-6f0a76a40b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"
Oct 30 19:00:59.203217 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.186274    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqbgp\" (UniqueName: \"kubernetes.io/projected/ccae37a5-4937-41df-9cae-6f0a76a40b38-kube-api-access-vqbgp\") pod \"maas-api-revoke-sub-b839a2d9-58p2k\" (UID: \"ccae37a5-4937-41df-9cae-6f0a76a40b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"
Oct 30 19:00:59.203217 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.186318    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b839a2d9-58p2k\" (UID: \"ccae37a5-4937-41df-9cae-6f0a76a40b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"
Oct 30 19:00:59.203217 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.198863    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b839a2d9-58p2k\" (UID: \"ccae37a5-4937-41df-9cae-6f0a76a40b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"
Oct 30 19:00:59.254169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.238953    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b839a2d9-58p2k\" (UID: \"ccae37a5-4937-41df-9cae-6f0a76a40b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"
Oct 30 19:00:59.325166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.324917    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:00:59.325166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.324974    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:00:59.334173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.330479    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqbgp\" (UniqueName: \"kubernetes.io/projected/ccae37a5-4937-41df-9cae-6f0a76a40b38-kube-api-access-vqbgp\") pod \"maas-api-revoke-sub-b839a2d9-58p2k\" (UID: \"ccae37a5-4937-41df-9cae-6f0a76a40b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"
Oct 30 19:00:59.375785 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.375727    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:59.377055 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.376976    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:59.411333 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.411288    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:59.411965 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.411595    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:00:59.412412 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.411624    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:00:59.454895 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.454781    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:59.457745 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.456987    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"
Oct 30 19:00:59.666672 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.666630    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:00:59.677311 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.677252    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 19:00:59.677545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.675343    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:00:59.677684 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.677662    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:00:59.862305 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.862269    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:00:59.862612 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.862583    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 19:00:59.946296 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.946248    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:59.954208 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:00:59.949063    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:01:00.116169 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:01:00.089987    6014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f582f52_06a8_4f9e_806d_06bf591b64e6.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:01:00.543338 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.458238    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:00.544672 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.544632    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:01:00.557235 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.544854    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:01:00.704953 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.559402    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:01:00.706121 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.559436    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:01:00.721204 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.560425    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:01:00.721204 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.560465    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:00.722917 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.571183    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:01:00.775594 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.775554    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:01:00.775850 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.775823    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:01:00.776044 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.776026    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:01:00.776200 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.776176    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 19:01:00.956736 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.956698    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:01:00.970165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.956994    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 19:01:00.972120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:00.972065    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:01.221224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.220882    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:01:01.316481 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.315235    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:01:01.316481 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.315311    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:01:01.455568 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.441222    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"]
Oct 30 19:01:01.471379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.468522    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"]
Oct 30 19:01:01.497473 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.496129    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:01.716613 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.716570    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:01.722430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.722345    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:01.842043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.722650    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:01:01.843227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.722686    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:01:01.865169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.744584    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:01:01.867245 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.867198    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pbcz\" (UniqueName: \"kubernetes.io/projected/9b7c2796-648a-48b1-b6a7-d83661404040-kube-api-access-7pbcz\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:01.867512 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.867489    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-dshm\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:01.867685 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.867669    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-home\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:01.867839 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.867816    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7c2796-648a-48b1-b6a7-d83661404040-tls-certs\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:01.868002 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.867984    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-model-cache\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:01.868108 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.868094    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-tmp-dir\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:01.894171 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.868943    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:01:02.233468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.874623    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:01:02.233468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.217922    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 19:01:02.233468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.012174    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-dshm\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:02.233468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.218050    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-home\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:02.233468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.218131    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7c2796-648a-48b1-b6a7-d83661404040-tls-certs\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:02.233468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.218263    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-model-cache\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:02.233468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.222186    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-tmp-dir\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:02.233468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.222686    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pbcz\" (UniqueName: \"kubernetes.io/projected/9b7c2796-648a-48b1-b6a7-d83661404040-kube-api-access-7pbcz\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:02.234755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.973495    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:02.254044 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.097693    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-dshm\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:02.254044 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.233711    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-home\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:02.254044 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.238730    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-model-cache\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:02.254044 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.239210    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-tmp-dir\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:02.254044 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:01.896966    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-66bdc0\"/\"distinct-2-w2-66bdc0-kserve-self-signed-certs\""
Oct 30 19:01:02.254044 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.217860    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 19:01:02.324180 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.316839    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7c2796-648a-48b1-b6a7-d83661404040-tls-certs\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:02.346703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.342044    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k" event={"ID":"ccae37a5-4937-41df-9cae-6f0a76a40b38","Type":"ContainerStarted","Data":"3307142c27aa6d8c97afb18c94e46ff12bcb85fa5d6c01f1f0b5a436042d77f8"}
Oct 30 19:01:02.346703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.345092    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:01:02.346703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.345161    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:01:02.346703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.345908    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:01:02.346703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.346174    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pbcz\" (UniqueName: \"kubernetes.io/projected/9b7c2796-648a-48b1-b6a7-d83661404040-kube-api-access-7pbcz\") pod \"distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") " pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:02.362226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.348982    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:01:02.362226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.355794    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 19:01:02.443818 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.443747    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"]
Oct 30 19:01:02.479191 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.477910    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:02.583163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.556019    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:01:02.715107 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.715034    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:02.715546 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.715511    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:02.723457 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.723347    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:01:02.723457 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.723415    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 19:01:02.726157 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.726112    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:01:02.821168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.764847    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:01:02.827049 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.826981    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:01:02.846613 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.846580    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:01:02.846882 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:02.846845    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 19:01:03.358801 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:03.358757    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k" event={"ID":"ccae37a5-4937-41df-9cae-6f0a76a40b38","Type":"ContainerStarted","Data":"c7425960d3034298df279b045ccfbefbfb5d9b230db90d49e129235c8cc3633c"}
Oct 30 19:01:03.414623 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:03.414589    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"]
Oct 30 19:01:03.429366 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:01:03.427304    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b7c2796_648a_48b1_b6a7_d83661404040.slice/crio-f3e92e2e8ce604cfb325237cb7337151671d2971a130e0b0db90ae8eec8b813f WatchSource:0}: Error finding container f3e92e2e8ce604cfb325237cb7337151671d2971a130e0b0db90ae8eec8b813f: Status 404 returned error can't find the container with id f3e92e2e8ce604cfb325237cb7337151671d2971a130e0b0db90ae8eec8b813f
Oct 30 19:01:03.492025 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:03.491395    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k" podStartSLOduration=5.49137459 podStartE2EDuration="5.49137459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:03.485017497 +0000 UTC m=+1302.128185110" watchObservedRunningTime="2026-10-30 19:01:03.49137459 +0000 UTC m=+1302.134542206"
Oct 30 19:01:03.567502 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:03.567456    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:01:03.567715 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:03.567524    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 19:01:03.758066 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:03.757705    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:03.758066 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:03.757764    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:03.845826 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:03.845707    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:01:03.845826 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:03.845766    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 19:01:03.913517 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:03.913483    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:01:03.915429 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:03.915405    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:01:03.925018 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:03.924949    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:01:04.376257 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.376222    6014 generic.go:356] "Generic (PLEG): container finished" podID="ccae37a5-4937-41df-9cae-6f0a76a40b38" containerID="c7425960d3034298df279b045ccfbefbfb5d9b230db90d49e129235c8cc3633c" exitCode=0
Oct 30 19:01:04.376923 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.376897    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k" event={"ID":"ccae37a5-4937-41df-9cae-6f0a76a40b38","Type":"ContainerDied","Data":"c7425960d3034298df279b045ccfbefbfb5d9b230db90d49e129235c8cc3633c"}
Oct 30 19:01:04.382163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.381523    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" event={"ID":"9b7c2796-648a-48b1-b6a7-d83661404040","Type":"ContainerStarted","Data":"2a644e0575eabebac5a50dcf71e57d9bf02f21b07f098456f255896862c10409"}
Oct 30 19:01:04.382361 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.382340    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" event={"ID":"9b7c2796-648a-48b1-b6a7-d83661404040","Type":"ContainerStarted","Data":"f3e92e2e8ce604cfb325237cb7337151671d2971a130e0b0db90ae8eec8b813f"}
Oct 30 19:01:04.496685 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.496213    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:04.584057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.580622    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:01:04.584057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.582281    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:01:04.584057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.583293    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:01:04.626611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.626507    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:01:04.630056 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.629992    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podStartSLOduration=3.62997099 podStartE2EDuration="3.62997099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:04.609050762 +0000 UTC m=+1303.252218378" watchObservedRunningTime="2026-10-30 19:01:04.62997099 +0000 UTC m=+1303.273138604"
Oct 30 19:01:04.680097 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.680064    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:01:04.994327 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.994250    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:01:04.994746 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.994729    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:01:04.995639 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:04.995621    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:01:05.052324 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.052294    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"]
Oct 30 19:01:05.062977 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.062945    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0981c105-9q92x"]
Oct 30 19:01:05.389704 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.389627    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:01:05.482325 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.482292    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489fa858-be00-432c-959d-b48f4e0d2ee4" path="/var/lib/kubelet/pods/489fa858-be00-432c-959d-b48f4e0d2ee4/volumes"
Oct 30 19:01:05.574341 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.574315    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"
Oct 30 19:01:05.660686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.660609    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-openshift-ca\") pod \"ccae37a5-4937-41df-9cae-6f0a76a40b38\" (UID: \"ccae37a5-4937-41df-9cae-6f0a76a40b38\") "
Oct 30 19:01:05.660841 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.660776    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccae37a5-4937-41df-9cae-6f0a76a40b38-kube-api-access-vqbgp\" (UniqueName: \"kubernetes.io/projected/ccae37a5-4937-41df-9cae-6f0a76a40b38-kube-api-access-vqbgp\") pod \"ccae37a5-4937-41df-9cae-6f0a76a40b38\" (UID: \"ccae37a5-4937-41df-9cae-6f0a76a40b38\") "
Oct 30 19:01:05.660841 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.660822    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-xks-ca\") pod \"ccae37a5-4937-41df-9cae-6f0a76a40b38\" (UID: \"ccae37a5-4937-41df-9cae-6f0a76a40b38\") "
Oct 30 19:01:05.661047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.661023    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-openshift-ca" pod "ccae37a5-4937-41df-9cae-6f0a76a40b38" (UID: "ccae37a5-4937-41df-9cae-6f0a76a40b38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:05.661486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.661459    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:05.663288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.663260    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccae37a5-4937-41df-9cae-6f0a76a40b38-kube-api-access-vqbgp" pod "ccae37a5-4937-41df-9cae-6f0a76a40b38" (UID: "ccae37a5-4937-41df-9cae-6f0a76a40b38"). InnerVolumeSpecName "kube-api-access-vqbgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:05.663288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.663276    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-xks-ca" pod "ccae37a5-4937-41df-9cae-6f0a76a40b38" (UID: "ccae37a5-4937-41df-9cae-6f0a76a40b38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:05.763218 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.763181    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqbgp\" (UniqueName: \"kubernetes.io/projected/ccae37a5-4937-41df-9cae-6f0a76a40b38-kube-api-access-vqbgp\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:05.763218 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:05.763215    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccae37a5-4937-41df-9cae-6f0a76a40b38-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:06.397311 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:06.397275    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"
Oct 30 19:01:06.397693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:06.397309    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k" event={"ID":"ccae37a5-4937-41df-9cae-6f0a76a40b38","Type":"ContainerDied","Data":"3307142c27aa6d8c97afb18c94e46ff12bcb85fa5d6c01f1f0b5a436042d77f8"}
Oct 30 19:01:06.397693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:06.397391    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3307142c27aa6d8c97afb18c94e46ff12bcb85fa5d6c01f1f0b5a436042d77f8"
Oct 30 19:01:10.048356 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:10.048320    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"]
Oct 30 19:01:10.097694 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:10.097661    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff5cc44-ltwpk"]
Oct 30 19:01:11.477382 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:11.477347    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb979b30-1d50-4178-872f-3bb08acd3518" path="/var/lib/kubelet/pods/cb979b30-1d50-4178-872f-3bb08acd3518/volumes"
Oct 30 19:01:12.492203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:12.492164    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:12.512053 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:12.512019    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:13.575549 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:13.575516    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:14.615167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:14.614270    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:14.916227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:14.892290    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:14.916227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:14.892737    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:01:14.928587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:14.927583    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.177061 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:15.158252    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.177061 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:15.159604    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.177061 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:15.173236    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.177061 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:15.174209    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.334659 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:15.294914    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:15.334659 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:15.294998    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.334659 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:15.295113    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.533204 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:15.533077    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.949168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:15.948481    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:16.079169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:15.957291    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:01:16.079169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:15.967246    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:01:16.235232 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.211270    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:16.310167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.309626    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:01:16.351166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.340417    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:01:16.351166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.340790    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:01:16.563254 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.562242    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": context deadline exceeded"
Oct 30 19:01:16.746165 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:01:16.740752    6014 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.258s"
Oct 30 19:01:16.746165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.741019    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:01:16.746165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.741192    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:01:16.746165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.741267    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 19:01:16.746165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.741324    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:01:16.746165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.741395    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:01:16.746165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.741511    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:01:16.746165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.741572    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:01:16.746165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.741635    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:16.799166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.763276    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:16.990224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.987977    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:01:16.990224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.988902    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:01:16.990224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:16.989130    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:01:17.256230 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:17.247724    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:17.318186 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:17.314239    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:17.467169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:17.463216    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:17.496403 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:17.463216    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:17.644167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:17.613232    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.124167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:18.122581    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.124167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:18.123599    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:01:18.124167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:18.123942    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:01:18.138262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:18.138219    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.210908 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:18.210863    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded"
Oct 30 19:01:18.262424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:18.262376    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.411962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:18.411869    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.676229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:18.660648    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.676229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:18.661487    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": context deadline exceeded"
Oct 30 19:01:18.676229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:18.663242    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.791612 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:18.687612    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"]
Oct 30 19:01:18.791612 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:18.768741    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cfbce5-ndmdc"]
Oct 30 19:01:19.086164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:19.067106    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Oct 30 19:01:19.254174 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:19.168616    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:01:19.255522 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:19.255483    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:01:19.268177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:19.259625    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:01:19.275540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:19.275481    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:01:19.446224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:19.445247    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:19.809055 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:19.808972    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05557c1-1499-4bec-bfa3-fbb27a93dec3" path="/var/lib/kubelet/pods/c05557c1-1499-4bec-bfa3-fbb27a93dec3/volumes"
Oct 30 19:01:19.913546 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:19.913493    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:19.914196 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:19.914164    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:01:19.948209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:19.943079    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:01:20.197175 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:20.175439    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:20.281373 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:20.280477    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:20.291215 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:20.291169    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:20.292645 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:20.292603    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:20.310496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:20.310467    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:01:20.328829 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:20.326062    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:01:20.339458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:20.337160    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:01:20.339458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:20.338080    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:20.339458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:20.338484    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:01:20.348067 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:20.347767    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 19:01:20.350422 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:20.350396    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:01:20.352430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:20.352411    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:01:21.621114 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.621085    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"]
Oct 30 19:01:21.621869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.621852    6014 cpu_manager.go:414] "containerMap: removing container" podUID="ccae37a5-4937-41df-9cae-6f0a76a40b38" containerName="revoke-keys"
Oct 30 19:01:21.621918 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.621870    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccae37a5-4937-41df-9cae-6f0a76a40b38" containerName="revoke-keys"
Oct 30 19:01:21.622046 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.622037    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccae37a5-4937-41df-9cae-6f0a76a40b38" containerName="revoke-keys"
Oct 30 19:01:21.635309 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.633742    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"
Oct 30 19:01:21.682751 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.682722    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"]
Oct 30 19:01:21.686373 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.686343    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:21.686524 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.686393    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:01:21.738939 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.738904    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bedce872-wjtb5\" (UID: \"9561525e-428b-4b15-8de4-9968b36566cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"
Oct 30 19:01:21.739151 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.739115    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bedce872-wjtb5\" (UID: \"9561525e-428b-4b15-8de4-9968b36566cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"
Oct 30 19:01:21.739221 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.739167    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf5fb\" (UniqueName: \"kubernetes.io/projected/9561525e-428b-4b15-8de4-9968b36566cb-kube-api-access-qf5fb\") pod \"maas-api-revoke-sub-bedce872-wjtb5\" (UID: \"9561525e-428b-4b15-8de4-9968b36566cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"
Oct 30 19:01:21.841300 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.841266    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bedce872-wjtb5\" (UID: \"9561525e-428b-4b15-8de4-9968b36566cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"
Oct 30 19:01:21.841499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.841420    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bedce872-wjtb5\" (UID: \"9561525e-428b-4b15-8de4-9968b36566cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"
Oct 30 19:01:21.841499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.841439    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf5fb\" (UniqueName: \"kubernetes.io/projected/9561525e-428b-4b15-8de4-9968b36566cb-kube-api-access-qf5fb\") pod \"maas-api-revoke-sub-bedce872-wjtb5\" (UID: \"9561525e-428b-4b15-8de4-9968b36566cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"
Oct 30 19:01:21.842013 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.841979    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bedce872-wjtb5\" (UID: \"9561525e-428b-4b15-8de4-9968b36566cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"
Oct 30 19:01:21.844373 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.844351    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bedce872-wjtb5\" (UID: \"9561525e-428b-4b15-8de4-9968b36566cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"
Oct 30 19:01:21.899578 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.899490    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf5fb\" (UniqueName: \"kubernetes.io/projected/9561525e-428b-4b15-8de4-9968b36566cb-kube-api-access-qf5fb\") pod \"maas-api-revoke-sub-bedce872-wjtb5\" (UID: \"9561525e-428b-4b15-8de4-9968b36566cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"
Oct 30 19:01:21.951778 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:21.951739    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"
Oct 30 19:01:22.043158 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:22.043026    6014 scope.go:122] "RemoveContainer" containerID="096a6655a1511016ad6de3457aab2f0b405c97a9626130c51b4dbb594fe41b1c"
Oct 30 19:01:22.061507 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:22.061480    6014 scope.go:122] "RemoveContainer" containerID="35702e58799122a7030f73e81085391d66e92db7ffcc113fc5b5f86c49fc0a1e"
Oct 30 19:01:22.350994 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:01:22.350948    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9561525e_428b_4b15_8de4_9968b36566cb.slice/crio-53cead538d8146f5c28e18a549b48bcc46a16e8aab7c71afd35392298adf845a WatchSource:0}: Error finding container 53cead538d8146f5c28e18a549b48bcc46a16e8aab7c71afd35392298adf845a: Status 404 returned error can't find the container with id 53cead538d8146f5c28e18a549b48bcc46a16e8aab7c71afd35392298adf845a
Oct 30 19:01:22.351430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:22.351392    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"]
Oct 30 19:01:23.264003 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:01:23.263963    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:01:23.357533 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:23.357492    6014 generic.go:356] "Generic (PLEG): container finished" podID="9561525e-428b-4b15-8de4-9968b36566cb" containerID="9a4fd2856512fc4a16f5bf53d5e4b0fcf21821571b35dc22014ee7c34f1423a1" exitCode=0
Oct 30 19:01:23.357720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:23.357570    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5" event={"ID":"9561525e-428b-4b15-8de4-9968b36566cb","Type":"ContainerDied","Data":"9a4fd2856512fc4a16f5bf53d5e4b0fcf21821571b35dc22014ee7c34f1423a1"}
Oct 30 19:01:23.357720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:23.357659    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5" event={"ID":"9561525e-428b-4b15-8de4-9968b36566cb","Type":"ContainerStarted","Data":"53cead538d8146f5c28e18a549b48bcc46a16e8aab7c71afd35392298adf845a"}
Oct 30 19:01:24.546393 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:24.546370    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"
Oct 30 19:01:24.583908 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:24.583876    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9561525e-428b-4b15-8de4-9968b36566cb-kube-api-access-qf5fb\" (UniqueName: \"kubernetes.io/projected/9561525e-428b-4b15-8de4-9968b36566cb-kube-api-access-qf5fb\") pod \"9561525e-428b-4b15-8de4-9968b36566cb\" (UID: \"9561525e-428b-4b15-8de4-9968b36566cb\") "
Oct 30 19:01:24.584109 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:24.583957    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-xks-ca\") pod \"9561525e-428b-4b15-8de4-9968b36566cb\" (UID: \"9561525e-428b-4b15-8de4-9968b36566cb\") "
Oct 30 19:01:24.584109 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:24.584022    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-openshift-ca\") pod \"9561525e-428b-4b15-8de4-9968b36566cb\" (UID: \"9561525e-428b-4b15-8de4-9968b36566cb\") "
Oct 30 19:01:24.584463 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:24.584425    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-openshift-ca" pod "9561525e-428b-4b15-8de4-9968b36566cb" (UID: "9561525e-428b-4b15-8de4-9968b36566cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:24.584693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:24.584673    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:24.586630 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:24.586603    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-xks-ca" pod "9561525e-428b-4b15-8de4-9968b36566cb" (UID: "9561525e-428b-4b15-8de4-9968b36566cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:24.586735 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:24.586672    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9561525e-428b-4b15-8de4-9968b36566cb-kube-api-access-qf5fb" pod "9561525e-428b-4b15-8de4-9968b36566cb" (UID: "9561525e-428b-4b15-8de4-9968b36566cb"). InnerVolumeSpecName "kube-api-access-qf5fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:24.685948 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:24.685865    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf5fb\" (UniqueName: \"kubernetes.io/projected/9561525e-428b-4b15-8de4-9968b36566cb-kube-api-access-qf5fb\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:24.685948 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:24.685892    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9561525e-428b-4b15-8de4-9968b36566cb-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:25.378021 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:25.377988    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"
Oct 30 19:01:25.378021 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:25.378014    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5" event={"ID":"9561525e-428b-4b15-8de4-9968b36566cb","Type":"ContainerDied","Data":"53cead538d8146f5c28e18a549b48bcc46a16e8aab7c71afd35392298adf845a"}
Oct 30 19:01:25.378281 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:25.378058    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53cead538d8146f5c28e18a549b48bcc46a16e8aab7c71afd35392298adf845a"
Oct 30 19:01:27.047219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:27.047176    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"]
Oct 30 19:01:27.096833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:27.096798    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02fad9f6-g4dv4"]
Oct 30 19:01:27.478015 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:27.477980    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d15b24-c4d3-40a2-9203-9db14250fa7c" path="/var/lib/kubelet/pods/b0d15b24-c4d3-40a2-9203-9db14250fa7c/volumes"
Oct 30 19:01:31.057913 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:31.057873    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"]
Oct 30 19:01:31.107188 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:31.107150    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1adf9edd-7w7ph"]
Oct 30 19:01:31.477464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:31.477432    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31dbdbfc-71a6-4f66-bfd9-68b3146b0f19" path="/var/lib/kubelet/pods/31dbdbfc-71a6-4f66-bfd9-68b3146b0f19/volumes"
Oct 30 19:01:35.125435 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.125396    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"]
Oct 30 19:01:35.126196 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.126177    6014 cpu_manager.go:414] "containerMap: removing container" podUID="9561525e-428b-4b15-8de4-9968b36566cb" containerName="revoke-keys"
Oct 30 19:01:35.126196 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.126196    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9561525e-428b-4b15-8de4-9968b36566cb" containerName="revoke-keys"
Oct 30 19:01:35.126397 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.126360    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="9561525e-428b-4b15-8de4-9968b36566cb" containerName="revoke-keys"
Oct 30 19:01:35.132259 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.132236    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.195088 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.195054    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"]
Oct 30 19:01:35.254862 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.254822    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrfr\" (UniqueName: \"kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-kube-api-access-dqrfr\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.255072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.254869    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.255072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.254900    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.255072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.255024    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.255288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.255178    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.255288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.255205    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.255288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.255237    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.255456 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.255419    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.255583 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.255567    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.357576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.357534    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.357737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.357582    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.357737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.357622    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.357737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.357700    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.357926 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.357786    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.357926 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.357848    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrfr\" (UniqueName: \"kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-kube-api-access-dqrfr\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.357926 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.357870    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.357926 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.357897    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.358133 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.357948    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.358133 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.358066    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.358321 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.358217    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.358321 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.358243    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.358486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.358461    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.358669 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.358649    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.360134 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.360110    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.360388 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.360362    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.415153 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.415052    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.415153 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.415129    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrfr\" (UniqueName: \"kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-kube-api-access-dqrfr\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.447932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.447898    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:35.903540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.903506    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"]
Oct 30 19:01:35.904039 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:01:35.904018    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f19a2ab_cc59_40f3_8d0f_b6ad0f76f12c.slice/crio-4efc25277db990acc91affe286d5f4858e5216bc1ee256e1ae836520a655e030 WatchSource:0}: Error finding container 4efc25277db990acc91affe286d5f4858e5216bc1ee256e1ae836520a655e030: Status 404 returned error can't find the container with id 4efc25277db990acc91affe286d5f4858e5216bc1ee256e1ae836520a655e030
Oct 30 19:01:35.906723 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.906666    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:01:35.906816 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.906774    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:01:35.906869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:35.906825    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:01:36.482166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:36.482047    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59" event={"ID":"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c","Type":"ContainerStarted","Data":"7502cc71ab581d311f1d876ff8a9d2e8d2d19dc4333c5088eeb192985480b7db"}
Oct 30 19:01:36.482166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:36.482110    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59" event={"ID":"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c","Type":"ContainerStarted","Data":"4efc25277db990acc91affe286d5f4858e5216bc1ee256e1ae836520a655e030"}
Oct 30 19:01:37.455174 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:37.454502    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:37.472707 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:37.472383    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:01:37.472707 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:37.472467    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59" podUID="0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 19:01:38.013989 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.013939    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59" podStartSLOduration=3.013918272 podStartE2EDuration="3.013918272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:36.666518448 +0000 UTC m=+1335.309686063" watchObservedRunningTime="2026-10-30 19:01:38.013918272 +0000 UTC m=+1336.657085887"
Oct 30 19:01:38.015555 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.015526    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"]
Oct 30 19:01:38.019955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.019934    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"
Oct 30 19:01:38.078953 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.078913    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"]
Oct 30 19:01:38.122080 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.122046    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:38.122292 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.122201    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:01:38.205977 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.205947    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7812ebe-v2b2h\" (UID: \"b1f619c5-b367-4e79-959f-1e022fc48c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"
Oct 30 19:01:38.206198 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.206083    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht57k\" (UniqueName: \"kubernetes.io/projected/b1f619c5-b367-4e79-959f-1e022fc48c23-kube-api-access-ht57k\") pod \"maas-api-revoke-sub-b7812ebe-v2b2h\" (UID: \"b1f619c5-b367-4e79-959f-1e022fc48c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"
Oct 30 19:01:38.206342 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.206325    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7812ebe-v2b2h\" (UID: \"b1f619c5-b367-4e79-959f-1e022fc48c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"
Oct 30 19:01:38.308116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.308027    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7812ebe-v2b2h\" (UID: \"b1f619c5-b367-4e79-959f-1e022fc48c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"
Oct 30 19:01:38.308310 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.308121    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht57k\" (UniqueName: \"kubernetes.io/projected/b1f619c5-b367-4e79-959f-1e022fc48c23-kube-api-access-ht57k\") pod \"maas-api-revoke-sub-b7812ebe-v2b2h\" (UID: \"b1f619c5-b367-4e79-959f-1e022fc48c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"
Oct 30 19:01:38.308356 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.308340    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7812ebe-v2b2h\" (UID: \"b1f619c5-b367-4e79-959f-1e022fc48c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"
Oct 30 19:01:38.308871 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.308851    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7812ebe-v2b2h\" (UID: \"b1f619c5-b367-4e79-959f-1e022fc48c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"
Oct 30 19:01:38.311028 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.311001    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7812ebe-v2b2h\" (UID: \"b1f619c5-b367-4e79-959f-1e022fc48c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"
Oct 30 19:01:38.372913 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.372877    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht57k\" (UniqueName: \"kubernetes.io/projected/b1f619c5-b367-4e79-959f-1e022fc48c23-kube-api-access-ht57k\") pod \"maas-api-revoke-sub-b7812ebe-v2b2h\" (UID: \"b1f619c5-b367-4e79-959f-1e022fc48c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"
Oct 30 19:01:38.464166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.461566    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:01:38.464166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.461627    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59" podUID="0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 19:01:38.635228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:38.635193    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"
Oct 30 19:01:39.075265 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.075233    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"]
Oct 30 19:01:39.080252 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:01:39.080213    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f619c5_b367_4e79_959f_1e022fc48c23.slice/crio-2a921f440348ee579fa2e8779d263877b2a456e19783c0ff4a523b55070acbd9 WatchSource:0}: Error finding container 2a921f440348ee579fa2e8779d263877b2a456e19783c0ff4a523b55070acbd9: Status 404 returned error can't find the container with id 2a921f440348ee579fa2e8779d263877b2a456e19783c0ff4a523b55070acbd9
Oct 30 19:01:39.470227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.456334    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:01:39.470227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.456396    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59" podUID="0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 19:01:39.483282 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.483254    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"]
Oct 30 19:01:39.492734 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.492703    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"
Oct 30 19:01:39.506964 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.506927    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h" event={"ID":"b1f619c5-b367-4e79-959f-1e022fc48c23","Type":"ContainerStarted","Data":"2a921f440348ee579fa2e8779d263877b2a456e19783c0ff4a523b55070acbd9"}
Oct 30 19:01:39.526411 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.526379    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"]
Oct 30 19:01:39.529842 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.529812    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34c04623-cqdtw\" (UID: \"14f26ae2-4b28-4b53-8619-9fab66457ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"
Oct 30 19:01:39.529996 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.529933    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34c04623-cqdtw\" (UID: \"14f26ae2-4b28-4b53-8619-9fab66457ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"
Oct 30 19:01:39.529996 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.529984    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zfmc\" (UniqueName: \"kubernetes.io/projected/14f26ae2-4b28-4b53-8619-9fab66457ea6-kube-api-access-7zfmc\") pod \"maas-api-revoke-sub-34c04623-cqdtw\" (UID: \"14f26ae2-4b28-4b53-8619-9fab66457ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"
Oct 30 19:01:39.631675 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.631633    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34c04623-cqdtw\" (UID: \"14f26ae2-4b28-4b53-8619-9fab66457ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"
Oct 30 19:01:39.631891 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.631747    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34c04623-cqdtw\" (UID: \"14f26ae2-4b28-4b53-8619-9fab66457ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"
Oct 30 19:01:39.631891 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.631812    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zfmc\" (UniqueName: \"kubernetes.io/projected/14f26ae2-4b28-4b53-8619-9fab66457ea6-kube-api-access-7zfmc\") pod \"maas-api-revoke-sub-34c04623-cqdtw\" (UID: \"14f26ae2-4b28-4b53-8619-9fab66457ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"
Oct 30 19:01:39.632383 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.632361    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34c04623-cqdtw\" (UID: \"14f26ae2-4b28-4b53-8619-9fab66457ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"
Oct 30 19:01:39.634216 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.634189    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34c04623-cqdtw\" (UID: \"14f26ae2-4b28-4b53-8619-9fab66457ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"
Oct 30 19:01:39.691364 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.691331    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zfmc\" (UniqueName: \"kubernetes.io/projected/14f26ae2-4b28-4b53-8619-9fab66457ea6-kube-api-access-7zfmc\") pod \"maas-api-revoke-sub-34c04623-cqdtw\" (UID: \"14f26ae2-4b28-4b53-8619-9fab66457ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"
Oct 30 19:01:39.812774 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:39.810938    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"
Oct 30 19:01:40.240551 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:40.240527    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"]
Oct 30 19:01:40.242056 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:01:40.242027    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f26ae2_4b28_4b53_8619_9fab66457ea6.slice/crio-ec7ba858bcbf9da219054bec457f6553c97bcd753fcf13a6fe24bc0a8f972bfd WatchSource:0}: Error finding container ec7ba858bcbf9da219054bec457f6553c97bcd753fcf13a6fe24bc0a8f972bfd: Status 404 returned error can't find the container with id ec7ba858bcbf9da219054bec457f6553c97bcd753fcf13a6fe24bc0a8f972bfd
Oct 30 19:01:40.469186 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:40.460267    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:01:40.469186 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:40.460333    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59" podUID="0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 19:01:40.517257 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:40.517133    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw" event={"ID":"14f26ae2-4b28-4b53-8619-9fab66457ea6","Type":"ContainerStarted","Data":"ec7ba858bcbf9da219054bec457f6553c97bcd753fcf13a6fe24bc0a8f972bfd"}
Oct 30 19:01:40.518913 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:40.518889    6014 generic.go:356] "Generic (PLEG): container finished" podID="b1f619c5-b367-4e79-959f-1e022fc48c23" containerID="4f3fbcd8724067fa6d6dad9bfc0c40353425934903ebd3198e895aecd34751fc" exitCode=0
Oct 30 19:01:40.519041 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:40.518945    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h" event={"ID":"b1f619c5-b367-4e79-959f-1e022fc48c23","Type":"ContainerDied","Data":"4f3fbcd8724067fa6d6dad9bfc0c40353425934903ebd3198e895aecd34751fc"}
Oct 30 19:01:41.472660 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.472624    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:01:41.473276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.472684    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59" podUID="0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 19:01:41.527994 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.527957    6014 generic.go:356] "Generic (PLEG): container finished" podID="14f26ae2-4b28-4b53-8619-9fab66457ea6" containerID="c8cdb4b536b8d01826b0de53c2276c5c5290e7df6b077ebcefc2e28b061c1749" exitCode=0
Oct 30 19:01:41.528200 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.528038    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw" event={"ID":"14f26ae2-4b28-4b53-8619-9fab66457ea6","Type":"ContainerDied","Data":"c8cdb4b536b8d01826b0de53c2276c5c5290e7df6b077ebcefc2e28b061c1749"}
Oct 30 19:01:41.784178 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.784127    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"
Oct 30 19:01:41.895270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.895218    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f619c5-b367-4e79-959f-1e022fc48c23-kube-api-access-ht57k\" (UniqueName: \"kubernetes.io/projected/b1f619c5-b367-4e79-959f-1e022fc48c23-kube-api-access-ht57k\") pod \"b1f619c5-b367-4e79-959f-1e022fc48c23\" (UID: \"b1f619c5-b367-4e79-959f-1e022fc48c23\") "
Oct 30 19:01:41.895460 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.895416    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-xks-ca\") pod \"b1f619c5-b367-4e79-959f-1e022fc48c23\" (UID: \"b1f619c5-b367-4e79-959f-1e022fc48c23\") "
Oct 30 19:01:41.895611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.895569    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-openshift-ca\") pod \"b1f619c5-b367-4e79-959f-1e022fc48c23\" (UID: \"b1f619c5-b367-4e79-959f-1e022fc48c23\") "
Oct 30 19:01:41.895874 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.895833    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-openshift-ca" pod "b1f619c5-b367-4e79-959f-1e022fc48c23" (UID: "b1f619c5-b367-4e79-959f-1e022fc48c23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:41.896928 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.896765    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:41.898081 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.898048    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f619c5-b367-4e79-959f-1e022fc48c23-kube-api-access-ht57k" pod "b1f619c5-b367-4e79-959f-1e022fc48c23" (UID: "b1f619c5-b367-4e79-959f-1e022fc48c23"). InnerVolumeSpecName "kube-api-access-ht57k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:41.898284 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.898263    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-xks-ca" pod "b1f619c5-b367-4e79-959f-1e022fc48c23" (UID: "b1f619c5-b367-4e79-959f-1e022fc48c23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:41.998699 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.998603    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht57k\" (UniqueName: \"kubernetes.io/projected/b1f619c5-b367-4e79-959f-1e022fc48c23-kube-api-access-ht57k\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:41.998699 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:41.998642    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f619c5-b367-4e79-959f-1e022fc48c23-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:42.051221 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.051187    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"]
Oct 30 19:01:42.055091 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.055064    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7f9644-lb98g"]
Oct 30 19:01:42.448596 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.448561    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:01:42.448765 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.448620    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59" podUID="0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 19:01:42.538095 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.538060    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h" event={"ID":"b1f619c5-b367-4e79-959f-1e022fc48c23","Type":"ContainerDied","Data":"2a921f440348ee579fa2e8779d263877b2a456e19783c0ff4a523b55070acbd9"}
Oct 30 19:01:42.538531 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.538116    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a921f440348ee579fa2e8779d263877b2a456e19783c0ff4a523b55070acbd9"
Oct 30 19:01:42.538531 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.538078    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"
Oct 30 19:01:42.726130 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.726103    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"
Oct 30 19:01:42.820091 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.820060    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-openshift-ca\") pod \"14f26ae2-4b28-4b53-8619-9fab66457ea6\" (UID: \"14f26ae2-4b28-4b53-8619-9fab66457ea6\") "
Oct 30 19:01:42.820308 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.820215    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f26ae2-4b28-4b53-8619-9fab66457ea6-kube-api-access-7zfmc\" (UniqueName: \"kubernetes.io/projected/14f26ae2-4b28-4b53-8619-9fab66457ea6-kube-api-access-7zfmc\") pod \"14f26ae2-4b28-4b53-8619-9fab66457ea6\" (UID: \"14f26ae2-4b28-4b53-8619-9fab66457ea6\") "
Oct 30 19:01:42.820365 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.820315    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-xks-ca\") pod \"14f26ae2-4b28-4b53-8619-9fab66457ea6\" (UID: \"14f26ae2-4b28-4b53-8619-9fab66457ea6\") "
Oct 30 19:01:42.820544 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.820520    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-openshift-ca" pod "14f26ae2-4b28-4b53-8619-9fab66457ea6" (UID: "14f26ae2-4b28-4b53-8619-9fab66457ea6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:42.821080 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.821057    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:42.822647 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.822618    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-xks-ca" pod "14f26ae2-4b28-4b53-8619-9fab66457ea6" (UID: "14f26ae2-4b28-4b53-8619-9fab66457ea6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:42.822744 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.822668    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f26ae2-4b28-4b53-8619-9fab66457ea6-kube-api-access-7zfmc" pod "14f26ae2-4b28-4b53-8619-9fab66457ea6" (UID: "14f26ae2-4b28-4b53-8619-9fab66457ea6"). InnerVolumeSpecName "kube-api-access-7zfmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:42.922392 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.922353    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f26ae2-4b28-4b53-8619-9fab66457ea6-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:42.922392 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:42.922385    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zfmc\" (UniqueName: \"kubernetes.io/projected/14f26ae2-4b28-4b53-8619-9fab66457ea6-kube-api-access-7zfmc\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:43.453677 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:43.453647    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:43.454183 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:43.454168    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:43.454795 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:43.454774    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:01:43.478279 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:43.478251    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a6b4585-367a-403f-be63-bd434498bc54" path="/var/lib/kubelet/pods/0a6b4585-367a-403f-be63-bd434498bc54/volumes"
Oct 30 19:01:43.547191 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:43.547131    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"
Oct 30 19:01:43.547599 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:43.547203    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw" event={"ID":"14f26ae2-4b28-4b53-8619-9fab66457ea6","Type":"ContainerDied","Data":"ec7ba858bcbf9da219054bec457f6553c97bcd753fcf13a6fe24bc0a8f972bfd"}
Oct 30 19:01:43.547599 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:43.547255    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec7ba858bcbf9da219054bec457f6553c97bcd753fcf13a6fe24bc0a8f972bfd"
Oct 30 19:01:46.025570 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:46.025537    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:01:48.435530 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:48.434252    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:48.667167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:48.666235    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:48.667498 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:48.667407    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:48.668177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:48.668129    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:48.984330 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:48.984273    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:48.988720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:48.988645    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:49.001298 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:49.001245    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:49.268222 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:49.265219    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:01:49.295167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:49.292241    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:49.435112 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:49.435061    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:49.439172 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:49.438883    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:01:49.700167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:49.670232    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": context deadline exceeded"
Oct 30 19:01:49.700167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:49.671041    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:49.700167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:49.673928    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:49.700167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:49.674339    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:01:49.700167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:49.675043    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:01:49.700167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:49.675096    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:01:50.057265 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:49.997903    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.057265 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:49.998254    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:01:50.061174 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:50.060584    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:01:50.061174 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:50.060761    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:01:50.280900 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:50.280855    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.291555 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:50.291521    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:01:50.299472 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:50.293494    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.304277 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:50.304247    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 19:01:50.449611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:50.449549    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.667161 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:50.657316    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:50.667161 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:50.657383    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.689920 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:50.681259    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.691955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:50.691913    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.708549 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:50.705195    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:01:51.017224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:51.014559    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:51.322643 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:51.322597    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:51.335121 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:51.325164    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:01:51.573260 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:51.544391    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:51.653728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:51.653679    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:01:51.708165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:51.704816    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:51.744169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:51.711799    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": context deadline exceeded"
Oct 30 19:01:52.068744 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:51.746620    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:01:52.245681 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:52.015405    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:52.272288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:52.245965    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:52.272288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:52.246028    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:52.332683 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:52.332568    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6" podUID="68730cc9-c11b-4699-a58e-792d376fe6d0" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.49:50052\" within 1s: context deadline exceeded"
Oct 30 19:01:52.354228 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:01:52.354195    6014 logging.go:55] [core] [Channel #1920 SubChannel #1922]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.49:50052", ServerName: "10.134.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.134.0.2:42186->10.134.0.49:50052: use of closed network connection"
Oct 30 19:01:52.379312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:52.379228    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:52.380867 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:52.380832    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:01:52.562468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:52.561680    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:52.684487 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:52.676299    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:52.769173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:52.758050    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:52.821185 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:01:52.816400    6014 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.07s"
Oct 30 19:01:53.143490 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:53.119551    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.143490 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:53.120055    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": context deadline exceeded"
Oct 30 19:01:53.242600 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:53.241601    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.397696 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:53.397562    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.398733 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:53.398692    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:01:53.419956 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:53.419917    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:53.420295 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:53.420267    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" podUID="945375bc-250a-4b52-8cfc-e405096baf26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.451207 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:53.449339    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.531772 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:53.531648    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.578240 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:53.578197    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:01:53.677672 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:53.676796    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:01:53.815048 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:53.814227    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:01:53.901187 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:01:53.894656    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:01:54.127166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:54.122239    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:54.127166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:54.122607    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:54.241973 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:54.241924    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:54.420366 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:54.420261    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:01:54.420937 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:54.420907    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:01:54.579286 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:54.579239    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:54.698226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:54.673309    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:54.894228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:54.880430    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:54.993226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:54.992701    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:55.138045 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.137997    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:55.150168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.138728    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": context deadline exceeded"
Oct 30 19:01:55.425298 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.414019    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:55.666951 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.425635    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:55.675177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.425703    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:01:55.740297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.426264    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:55.741071 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.690923    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:01:55.785365 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.785265    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:01:55.785365 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.785344    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:01:55.785592 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.785399    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:01:55.785592 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.785447    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:01:55.809702 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.809506    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:01:55.809702 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.809571    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 19:01:55.813191 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.812854    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:01:55.813785 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:55.813280    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:01:57.960203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:57.960165    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"]
Oct 30 19:01:57.960898 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:57.960883    6014 cpu_manager.go:414] "containerMap: removing container" podUID="b1f619c5-b367-4e79-959f-1e022fc48c23" containerName="revoke-keys"
Oct 30 19:01:57.960941 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:57.960899    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f619c5-b367-4e79-959f-1e022fc48c23" containerName="revoke-keys"
Oct 30 19:01:57.960941 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:57.960929    6014 cpu_manager.go:414] "containerMap: removing container" podUID="14f26ae2-4b28-4b53-8619-9fab66457ea6" containerName="revoke-keys"
Oct 30 19:01:57.960941 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:57.960934    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f26ae2-4b28-4b53-8619-9fab66457ea6" containerName="revoke-keys"
Oct 30 19:01:57.961061 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:57.961052    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f26ae2-4b28-4b53-8619-9fab66457ea6" containerName="revoke-keys"
Oct 30 19:01:57.961095 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:57.961071    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f619c5-b367-4e79-959f-1e022fc48c23" containerName="revoke-keys"
Oct 30 19:01:57.966099 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:57.966083    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"
Oct 30 19:01:58.019540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.019511    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"]
Oct 30 19:01:58.072466 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.072430    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:01:58.073447 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.072655    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:58.088286 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.088254    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60b3957d-n9wtr\" (UID: \"0918e5bb-4103-4713-9f14-0848c3e37ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"
Oct 30 19:01:58.088445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.088423    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m27m\" (UniqueName: \"kubernetes.io/projected/0918e5bb-4103-4713-9f14-0848c3e37ab3-kube-api-access-8m27m\") pod \"maas-api-revoke-sub-60b3957d-n9wtr\" (UID: \"0918e5bb-4103-4713-9f14-0848c3e37ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"
Oct 30 19:01:58.088498 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.088481    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60b3957d-n9wtr\" (UID: \"0918e5bb-4103-4713-9f14-0848c3e37ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"
Oct 30 19:01:58.190041 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.190006    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60b3957d-n9wtr\" (UID: \"0918e5bb-4103-4713-9f14-0848c3e37ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"
Oct 30 19:01:58.190228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.190192    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m27m\" (UniqueName: \"kubernetes.io/projected/0918e5bb-4103-4713-9f14-0848c3e37ab3-kube-api-access-8m27m\") pod \"maas-api-revoke-sub-60b3957d-n9wtr\" (UID: \"0918e5bb-4103-4713-9f14-0848c3e37ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"
Oct 30 19:01:58.190302 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.190288    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60b3957d-n9wtr\" (UID: \"0918e5bb-4103-4713-9f14-0848c3e37ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"
Oct 30 19:01:58.190917 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.190894    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60b3957d-n9wtr\" (UID: \"0918e5bb-4103-4713-9f14-0848c3e37ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"
Oct 30 19:01:58.192493 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.192470    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60b3957d-n9wtr\" (UID: \"0918e5bb-4103-4713-9f14-0848c3e37ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"
Oct 30 19:01:58.247475 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.247398    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m27m\" (UniqueName: \"kubernetes.io/projected/0918e5bb-4103-4713-9f14-0848c3e37ab3-kube-api-access-8m27m\") pod \"maas-api-revoke-sub-60b3957d-n9wtr\" (UID: \"0918e5bb-4103-4713-9f14-0848c3e37ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"
Oct 30 19:01:58.280484 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.280452    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"
Oct 30 19:01:58.661079 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.661048    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"]
Oct 30 19:01:58.661749 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:01:58.661721    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0918e5bb_4103_4713_9f14_0848c3e37ab3.slice/crio-c78c9c7c17776a2e44cf6f023a237a66cdbf0b4cf3ae212f1f13b0207856caa4 WatchSource:0}: Error finding container c78c9c7c17776a2e44cf6f023a237a66cdbf0b4cf3ae212f1f13b0207856caa4: Status 404 returned error can't find the container with id c78c9c7c17776a2e44cf6f023a237a66cdbf0b4cf3ae212f1f13b0207856caa4
Oct 30 19:01:58.859830 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:58.859791    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr" event={"ID":"0918e5bb-4103-4713-9f14-0848c3e37ab3","Type":"ContainerStarted","Data":"c78c9c7c17776a2e44cf6f023a237a66cdbf0b4cf3ae212f1f13b0207856caa4"}
Oct 30 19:01:59.869860 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:59.869829    6014 generic.go:356] "Generic (PLEG): container finished" podID="0918e5bb-4103-4713-9f14-0848c3e37ab3" containerID="e9ca279e8df5f6de9e86085edb875c130c46db3a22f52ab87b1cc49325c1c342" exitCode=0
Oct 30 19:01:59.870288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:01:59.869921    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr" event={"ID":"0918e5bb-4103-4713-9f14-0848c3e37ab3","Type":"ContainerDied","Data":"e9ca279e8df5f6de9e86085edb875c130c46db3a22f52ab87b1cc49325c1c342"}
Oct 30 19:02:01.069115 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:01.069073    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"
Oct 30 19:02:01.136445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:01.136356    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-openshift-ca\") pod \"0918e5bb-4103-4713-9f14-0848c3e37ab3\" (UID: \"0918e5bb-4103-4713-9f14-0848c3e37ab3\") "
Oct 30 19:02:01.136599 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:01.136492    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0918e5bb-4103-4713-9f14-0848c3e37ab3-kube-api-access-8m27m\" (UniqueName: \"kubernetes.io/projected/0918e5bb-4103-4713-9f14-0848c3e37ab3-kube-api-access-8m27m\") pod \"0918e5bb-4103-4713-9f14-0848c3e37ab3\" (UID: \"0918e5bb-4103-4713-9f14-0848c3e37ab3\") "
Oct 30 19:02:01.136599 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:01.136530    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-xks-ca\") pod \"0918e5bb-4103-4713-9f14-0848c3e37ab3\" (UID: \"0918e5bb-4103-4713-9f14-0848c3e37ab3\") "
Oct 30 19:02:01.136760 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:01.136737    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-openshift-ca" pod "0918e5bb-4103-4713-9f14-0848c3e37ab3" (UID: "0918e5bb-4103-4713-9f14-0848c3e37ab3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:01.137156 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:01.137120    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:01.138760 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:01.138734    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-xks-ca" pod "0918e5bb-4103-4713-9f14-0848c3e37ab3" (UID: "0918e5bb-4103-4713-9f14-0848c3e37ab3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:01.138849 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:01.138782    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0918e5bb-4103-4713-9f14-0848c3e37ab3-kube-api-access-8m27m" pod "0918e5bb-4103-4713-9f14-0848c3e37ab3" (UID: "0918e5bb-4103-4713-9f14-0848c3e37ab3"). InnerVolumeSpecName "kube-api-access-8m27m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:01.238847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:01.238808    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m27m\" (UniqueName: \"kubernetes.io/projected/0918e5bb-4103-4713-9f14-0848c3e37ab3-kube-api-access-8m27m\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:01.238847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:01.238840    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0918e5bb-4103-4713-9f14-0848c3e37ab3-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:01.893607 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:01.893575    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"
Oct 30 19:02:01.893607 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:01.893596    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr" event={"ID":"0918e5bb-4103-4713-9f14-0848c3e37ab3","Type":"ContainerDied","Data":"c78c9c7c17776a2e44cf6f023a237a66cdbf0b4cf3ae212f1f13b0207856caa4"}
Oct 30 19:02:01.893832 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:01.893669    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c78c9c7c17776a2e44cf6f023a237a66cdbf0b4cf3ae212f1f13b0207856caa4"
Oct 30 19:02:04.299076 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.299045    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"]
Oct 30 19:02:04.299818 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.299803    6014 cpu_manager.go:414] "containerMap: removing container" podUID="0918e5bb-4103-4713-9f14-0848c3e37ab3" containerName="revoke-keys"
Oct 30 19:02:04.299875 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.299819    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0918e5bb-4103-4713-9f14-0848c3e37ab3" containerName="revoke-keys"
Oct 30 19:02:04.299934 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.299925    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0918e5bb-4103-4713-9f14-0848c3e37ab3" containerName="revoke-keys"
Oct 30 19:02:04.303766 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.303748    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.364169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.364123    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"]
Oct 30 19:02:04.391103 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.391068    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.391315 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.391134    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.391315 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.391266    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311a8fef-9742-4418-b638-ee0f8a474764-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.391315 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.391309    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311a8fef-9742-4418-b638-ee0f8a474764-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.391488 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.391367    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.391488 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.391474    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9gd\" (UniqueName: \"kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-kube-api-access-rm9gd\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.391591 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.391505    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.391647 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.391637    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.391705 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.391671    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.493389 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.493353    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9gd\" (UniqueName: \"kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-kube-api-access-rm9gd\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.493587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.493407    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.493587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.493523    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.493587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.493562    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.493755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.493731    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.493812 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.493795    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.493861 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.493830    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311a8fef-9742-4418-b638-ee0f8a474764-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.493910 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.493795    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.493910 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.493878    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311a8fef-9742-4418-b638-ee0f8a474764-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.494020 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.493964    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.494020 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.493964    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.494126 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.494078    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.494306 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.494278    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.494683 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.494630    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311a8fef-9742-4418-b638-ee0f8a474764-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.496163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.496122    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.496417 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.496399    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311a8fef-9742-4418-b638-ee0f8a474764-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.551052 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.550979    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.551245 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.551156    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9gd\" (UniqueName: \"kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-kube-api-access-rm9gd\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:04.619078 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:04.619045    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:05.066114 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:05.066078    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"]
Oct 30 19:02:05.093920 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:02:05.093891    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311a8fef_9742_4418_b638_ee0f8a474764.slice/crio-18a4c2ee52b2ef4f1363865ec335aaed51c7261f36c1e6d8abd6eaf211f8e3dc WatchSource:0}: Error finding container 18a4c2ee52b2ef4f1363865ec335aaed51c7261f36c1e6d8abd6eaf211f8e3dc: Status 404 returned error can't find the container with id 18a4c2ee52b2ef4f1363865ec335aaed51c7261f36c1e6d8abd6eaf211f8e3dc
Oct 30 19:02:05.096669 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:05.096633    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:02:05.096744 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:05.096709    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:02:05.096777 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:05.096740    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:02:05.934129 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:05.934093    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8" event={"ID":"311a8fef-9742-4418-b638-ee0f8a474764","Type":"ContainerStarted","Data":"9371167ab8afbed92a3f023e8d5d5bcc8e8c4a3480dec6a40aee528f1df5b1c6"}
Oct 30 19:02:05.934549 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:05.934177    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8" event={"ID":"311a8fef-9742-4418-b638-ee0f8a474764","Type":"ContainerStarted","Data":"18a4c2ee52b2ef4f1363865ec335aaed51c7261f36c1e6d8abd6eaf211f8e3dc"}
Oct 30 19:02:06.075731 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:06.075677    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8" podStartSLOduration=2.075661466 podStartE2EDuration="2.075661466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:06.069804024 +0000 UTC m=+1364.712971671" watchObservedRunningTime="2026-10-30 19:02:06.075661466 +0000 UTC m=+1364.718829079"
Oct 30 19:02:06.619525 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:06.619485    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:06.621046 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:06.621019    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 19:02:06.621217 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:06.621078    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8" podUID="311a8fef-9742-4418-b638-ee0f8a474764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 19:02:07.619795 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:07.619761    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 19:02:07.620237 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:07.619837    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8" podUID="311a8fef-9742-4418-b638-ee0f8a474764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 19:02:08.619596 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.619561    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 19:02:08.619783 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.619625    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8" podUID="311a8fef-9742-4418-b638-ee0f8a474764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 19:02:08.685944 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.685904    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"]
Oct 30 19:02:08.691015 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.690996    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"
Oct 30 19:02:08.746863 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.746832    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"]
Oct 30 19:02:08.786564 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.786528    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:02:08.788652 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.788624    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q7fj\" (UniqueName: \"kubernetes.io/projected/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-kube-api-access-5q7fj\") pod \"maas-api-revoke-sub-c55a831a-8fnnx\" (UID: \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"
Oct 30 19:02:08.788804 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.788713    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c55a831a-8fnnx\" (UID: \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"
Oct 30 19:02:08.788804 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.788758    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c55a831a-8fnnx\" (UID: \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"
Oct 30 19:02:08.790773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.790751    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:02:08.890040 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.889949    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c55a831a-8fnnx\" (UID: \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"
Oct 30 19:02:08.890040 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.889997    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c55a831a-8fnnx\" (UID: \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"
Oct 30 19:02:08.890271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.890248    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q7fj\" (UniqueName: \"kubernetes.io/projected/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-kube-api-access-5q7fj\") pod \"maas-api-revoke-sub-c55a831a-8fnnx\" (UID: \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"
Oct 30 19:02:08.890773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.890753    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c55a831a-8fnnx\" (UID: \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"
Oct 30 19:02:08.892637 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.892606    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c55a831a-8fnnx\" (UID: \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"
Oct 30 19:02:08.946785 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:08.946752    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q7fj\" (UniqueName: \"kubernetes.io/projected/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-kube-api-access-5q7fj\") pod \"maas-api-revoke-sub-c55a831a-8fnnx\" (UID: \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"
Oct 30 19:02:09.006017 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:09.005972    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"
Oct 30 19:02:09.416681 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:09.416648    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"]
Oct 30 19:02:09.620578 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:09.620542    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 19:02:09.620760 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:09.620604    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8" podUID="311a8fef-9742-4418-b638-ee0f8a474764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 19:02:09.989269 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:09.989237    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx" event={"ID":"f5962d0b-f82e-42d7-ae83-a482e5f32ef1","Type":"ContainerStarted","Data":"e854df9d41a3ae34e456e73f6c447663db1143e4981de9185897a71440cc4d2e"}
Oct 30 19:02:09.989749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:09.989725    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx" event={"ID":"f5962d0b-f82e-42d7-ae83-a482e5f32ef1","Type":"ContainerStarted","Data":"1bd57df76b06feec3f00bbed6e849684e7a54e462a89b2d6b8a782d07589b090"}
Oct 30 19:02:10.103107 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:10.103059    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx" podStartSLOduration=2.103043545 podStartE2EDuration="2.103043545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:10.100701181 +0000 UTC m=+1368.743868794" watchObservedRunningTime="2026-10-30 19:02:10.103043545 +0000 UTC m=+1368.746211158"
Oct 30 19:02:10.619933 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:10.619898    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 19:02:10.620073 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:10.619956    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8" podUID="311a8fef-9742-4418-b638-ee0f8a474764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 19:02:10.998092 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:10.998001    6014 generic.go:356] "Generic (PLEG): container finished" podID="f5962d0b-f82e-42d7-ae83-a482e5f32ef1" containerID="e854df9d41a3ae34e456e73f6c447663db1143e4981de9185897a71440cc4d2e" exitCode=0
Oct 30 19:02:10.998520 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:10.998088    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx" event={"ID":"f5962d0b-f82e-42d7-ae83-a482e5f32ef1","Type":"ContainerDied","Data":"e854df9d41a3ae34e456e73f6c447663db1143e4981de9185897a71440cc4d2e"}
Oct 30 19:02:11.620200 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:11.620166    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 19:02:11.620400 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:11.620226    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8" podUID="311a8fef-9742-4418-b638-ee0f8a474764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 19:02:12.206835 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:12.206810    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"
Oct 30 19:02:12.343077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:12.343026    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-openshift-ca\") pod \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\" (UID: \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\") "
Oct 30 19:02:12.343408 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:12.343218    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-xks-ca\") pod \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\" (UID: \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\") "
Oct 30 19:02:12.343408 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:12.343324    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-kube-api-access-5q7fj\" (UniqueName: \"kubernetes.io/projected/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-kube-api-access-5q7fj\") pod \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\" (UID: \"f5962d0b-f82e-42d7-ae83-a482e5f32ef1\") "
Oct 30 19:02:12.343988 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:12.343960    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-openshift-ca" pod "f5962d0b-f82e-42d7-ae83-a482e5f32ef1" (UID: "f5962d0b-f82e-42d7-ae83-a482e5f32ef1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:12.344533 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:12.344508    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:12.346223 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:12.346184    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-kube-api-access-5q7fj" pod "f5962d0b-f82e-42d7-ae83-a482e5f32ef1" (UID: "f5962d0b-f82e-42d7-ae83-a482e5f32ef1"). InnerVolumeSpecName "kube-api-access-5q7fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:12.346458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:12.346434    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-xks-ca" pod "f5962d0b-f82e-42d7-ae83-a482e5f32ef1" (UID: "f5962d0b-f82e-42d7-ae83-a482e5f32ef1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:12.449825 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:12.446306    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:12.449825 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:12.446342    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q7fj\" (UniqueName: \"kubernetes.io/projected/f5962d0b-f82e-42d7-ae83-a482e5f32ef1-kube-api-access-5q7fj\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:12.628281 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:12.628071    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:12.629349 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:12.629329    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:12.629980 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:12.629958    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:02:13.016714 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:13.016638    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"
Oct 30 19:02:13.016714 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:13.016664    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx" event={"ID":"f5962d0b-f82e-42d7-ae83-a482e5f32ef1","Type":"ContainerDied","Data":"1bd57df76b06feec3f00bbed6e849684e7a54e462a89b2d6b8a782d07589b090"}
Oct 30 19:02:13.016714 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:13.016710    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bd57df76b06feec3f00bbed6e849684e7a54e462a89b2d6b8a782d07589b090"
Oct 30 19:02:16.505624 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.505585    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"]
Oct 30 19:02:16.506576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.506557    6014 cpu_manager.go:414] "containerMap: removing container" podUID="f5962d0b-f82e-42d7-ae83-a482e5f32ef1" containerName="revoke-keys"
Oct 30 19:02:16.506627 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.506577    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5962d0b-f82e-42d7-ae83-a482e5f32ef1" containerName="revoke-keys"
Oct 30 19:02:16.506717 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.506708    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5962d0b-f82e-42d7-ae83-a482e5f32ef1" containerName="revoke-keys"
Oct 30 19:02:16.510639 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.510622    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"
Oct 30 19:02:16.567297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.567259    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"]
Oct 30 19:02:16.610701 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.610671    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:02:16.610902 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.610800    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:02:16.620036 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.619999    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95358bf7-rrxbc\" (UID: \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"
Oct 30 19:02:16.620233 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.620058    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95358bf7-rrxbc\" (UID: \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"
Oct 30 19:02:16.620320 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.620226    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqtkt\" (UniqueName: \"kubernetes.io/projected/0062b2f7-32de-4f15-9c6c-7e50fe61584d-kube-api-access-qqtkt\") pod \"maas-api-revoke-sub-95358bf7-rrxbc\" (UID: \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"
Oct 30 19:02:16.722602 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.722566    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95358bf7-rrxbc\" (UID: \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"
Oct 30 19:02:16.722782 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.722632    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95358bf7-rrxbc\" (UID: \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"
Oct 30 19:02:16.722782 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.722748    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqtkt\" (UniqueName: \"kubernetes.io/projected/0062b2f7-32de-4f15-9c6c-7e50fe61584d-kube-api-access-qqtkt\") pod \"maas-api-revoke-sub-95358bf7-rrxbc\" (UID: \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"
Oct 30 19:02:16.723335 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.723310    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95358bf7-rrxbc\" (UID: \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"
Oct 30 19:02:16.725302 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.725285    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95358bf7-rrxbc\" (UID: \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"
Oct 30 19:02:16.779738 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.779660    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqtkt\" (UniqueName: \"kubernetes.io/projected/0062b2f7-32de-4f15-9c6c-7e50fe61584d-kube-api-access-qqtkt\") pod \"maas-api-revoke-sub-95358bf7-rrxbc\" (UID: \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"
Oct 30 19:02:16.825624 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:16.825589    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"
Oct 30 19:02:17.211263 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:17.211227    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"]
Oct 30 19:02:17.216311 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:02:17.216275    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0062b2f7_32de_4f15_9c6c_7e50fe61584d.slice/crio-0e4dfde03c9316d3b6380cd79d0b6c8a3c02bebc4fab6820c59e2d568b9bcc9c WatchSource:0}: Error finding container 0e4dfde03c9316d3b6380cd79d0b6c8a3c02bebc4fab6820c59e2d568b9bcc9c: Status 404 returned error can't find the container with id 0e4dfde03c9316d3b6380cd79d0b6c8a3c02bebc4fab6820c59e2d568b9bcc9c
Oct 30 19:02:18.063008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:18.062973    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc" event={"ID":"0062b2f7-32de-4f15-9c6c-7e50fe61584d","Type":"ContainerStarted","Data":"44a012876b307fee793e9abaf4c10769f755cc59acbcf98f7c7732f8a625055e"}
Oct 30 19:02:18.063444 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:18.063031    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc" event={"ID":"0062b2f7-32de-4f15-9c6c-7e50fe61584d","Type":"ContainerStarted","Data":"0e4dfde03c9316d3b6380cd79d0b6c8a3c02bebc4fab6820c59e2d568b9bcc9c"}
Oct 30 19:02:18.176448 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:18.176386    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc" podStartSLOduration=2.176367289 podStartE2EDuration="2.176367289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:18.175643236 +0000 UTC m=+1376.818810849" watchObservedRunningTime="2026-10-30 19:02:18.176367289 +0000 UTC m=+1376.819534907"
Oct 30 19:02:19.072939 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:19.072888    6014 generic.go:356] "Generic (PLEG): container finished" podID="0062b2f7-32de-4f15-9c6c-7e50fe61584d" containerID="44a012876b307fee793e9abaf4c10769f755cc59acbcf98f7c7732f8a625055e" exitCode=0
Oct 30 19:02:19.073438 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:19.073203    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc" event={"ID":"0062b2f7-32de-4f15-9c6c-7e50fe61584d","Type":"ContainerDied","Data":"44a012876b307fee793e9abaf4c10769f755cc59acbcf98f7c7732f8a625055e"}
Oct 30 19:02:20.250759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:20.250731    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"
Oct 30 19:02:20.385891 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:20.385803    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-xks-ca\") pod \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\" (UID: \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\") "
Oct 30 19:02:20.386066 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:20.385925    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0062b2f7-32de-4f15-9c6c-7e50fe61584d-kube-api-access-qqtkt\" (UniqueName: \"kubernetes.io/projected/0062b2f7-32de-4f15-9c6c-7e50fe61584d-kube-api-access-qqtkt\") pod \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\" (UID: \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\") "
Oct 30 19:02:20.386066 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:20.386048    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-openshift-ca\") pod \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\" (UID: \"0062b2f7-32de-4f15-9c6c-7e50fe61584d\") "
Oct 30 19:02:20.386976 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:20.386644    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-openshift-ca" pod "0062b2f7-32de-4f15-9c6c-7e50fe61584d" (UID: "0062b2f7-32de-4f15-9c6c-7e50fe61584d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:20.386976 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:20.386945    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:20.389253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:20.389211    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-xks-ca" pod "0062b2f7-32de-4f15-9c6c-7e50fe61584d" (UID: "0062b2f7-32de-4f15-9c6c-7e50fe61584d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:20.390212 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:20.390184    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0062b2f7-32de-4f15-9c6c-7e50fe61584d-kube-api-access-qqtkt" pod "0062b2f7-32de-4f15-9c6c-7e50fe61584d" (UID: "0062b2f7-32de-4f15-9c6c-7e50fe61584d"). InnerVolumeSpecName "kube-api-access-qqtkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:20.488243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:20.488209    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0062b2f7-32de-4f15-9c6c-7e50fe61584d-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:20.488243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:20.488240    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqtkt\" (UniqueName: \"kubernetes.io/projected/0062b2f7-32de-4f15-9c6c-7e50fe61584d-kube-api-access-qqtkt\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:21.089511 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:21.089484    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"
Oct 30 19:02:21.089686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:21.089516    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc" event={"ID":"0062b2f7-32de-4f15-9c6c-7e50fe61584d","Type":"ContainerDied","Data":"0e4dfde03c9316d3b6380cd79d0b6c8a3c02bebc4fab6820c59e2d568b9bcc9c"}
Oct 30 19:02:21.089686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:21.089566    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e4dfde03c9316d3b6380cd79d0b6c8a3c02bebc4fab6820c59e2d568b9bcc9c"
Oct 30 19:02:22.139638 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:22.139592    6014 scope.go:122] "RemoveContainer" containerID="f890389e400dc3dbdbeea176323208285433aa216ab779d54756615bb940b9ee"
Oct 30 19:02:22.337909 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:22.337473    6014 scope.go:122] "RemoveContainer" containerID="1f6dc833490352f76bc55866fcb063820003f98febc82321d5fc6fd29fe1cdf4"
Oct 30 19:02:22.394390 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:22.393831    6014 scope.go:122] "RemoveContainer" containerID="047d67fd961295dd3c2f7fdd74cea0410a0393f8bb86df40b6376cff6b954113"
Oct 30 19:02:22.586124 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:22.586079    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:02:22.622412 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:22.621613    6014 scope.go:122] "RemoveContainer" containerID="13d399c2a57de5dbfa31975945971d837c4ca7c666129062307b3d5a7921856d"
Oct 30 19:02:22.835944 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:22.835696    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:22.975128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:22.974974    6014 scope.go:122] "RemoveContainer" containerID="14dde87cce76c56a5e641cfeb26114b48b83b813f7056f6363ec739875469f89"
Oct 30 19:02:23.020699 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:23.020652    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:23.077350 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:23.077305    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:23.078162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:23.078110    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:02:23.290453 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:23.290384    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:23.359515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:23.359471    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:23.409028 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:23.407344    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:02:23.514777 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:23.514278    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:23.599162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:23.598248    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:02:23.599162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:23.598880    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:02:23.707631 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:23.707550    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:23.728100 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:02:23.728064    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:02:23.878727 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:23.843252    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:02:23.878727 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:23.843938    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:02:24.064167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:24.051899    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:02:24.079799 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:24.078298    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.079799 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:24.078806    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.169236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:24.167050    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:02:24.169236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:24.168267    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:02:24.169236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:24.168329    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:02:24.393168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:24.392978    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.434224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:24.398262    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:02:24.450175 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:24.439997    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.450175 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:24.441405    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 19:02:25.005367 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:24.952283    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Oct 30 19:02:25.005367 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:24.987332    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:02:25.368238 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:25.182388    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:02:25.368238 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:25.294260    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:02:25.368238 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:25.328079    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:02:25.397180 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:25.394308    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded"
Oct 30 19:02:25.433180 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:25.416914    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:25.504164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:25.501339    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:25.873284 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:25.873238    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:02:25.896643 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:25.896583    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz" podUID="2d7fbb25-1611-4efd-83fb-c636fdc232ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:25.953084 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:25.953036    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jg64x" podUID="79f5eaf5-cfd1-44df-878c-70c7798c2c12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.090164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:26.088672    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:02:26.352231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:26.327245    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.379168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:26.356725    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs" podUID="7c1d3787-e640-4027-b267-88a3c8a19873" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.54:50052\" within 1s: context deadline exceeded"
Oct 30 19:02:26.526165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:26.523328    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.575167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:26.547356    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:02:26.835728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:26.688642    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.835728 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:02:26.783367    6014 logging.go:55] [core] [Channel #2039 SubChannel #2040]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.54:50052", ServerName: "10.134.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.134.0.2:58004->10.134.0.54:50052: use of closed network connection"
Oct 30 19:02:26.836648 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:26.836601    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.846353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:26.842216    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.980619 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:26.980577    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:02:27.004654 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:25.994569    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.072752 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:27.071715    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z6rtg" podUID="e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.072752 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:27.072739    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz" podUID="2d7fbb25-1611-4efd-83fb-c636fdc232ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.125856 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:27.125744    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.244293 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:02:27.228618    6014 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.473s"
Oct 30 19:02:27.244674 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:27.244651    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 19:02:27.329389 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:27.329340    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.366305 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:27.347286    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67kkz"
Oct 30 19:02:27.568225 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:27.567405    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.820231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:27.811504    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.892265 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:27.879254    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.916538 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:27.916483    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:02:27.916885 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:27.916832    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" podUID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.920185 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:27.919194    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:28.101228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:28.025321    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:02:28.118681 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:28.037069    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:02:28.177290 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:28.177224    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:28.259100 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:28.259056    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:28.307391 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:28.305870    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:02:28.308678 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:28.308652    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:02:28.310112 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:28.309706    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:02:28.310282 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:28.310118    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:02:28.310726 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:28.310702    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:02:28.311175 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:28.311157    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:02:28.317404 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:28.317385    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 19:02:28.319939 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:28.319924    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:02:29.233101 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.233068    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"]
Oct 30 19:02:29.233974 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.233956    6014 cpu_manager.go:414] "containerMap: removing container" podUID="0062b2f7-32de-4f15-9c6c-7e50fe61584d" containerName="revoke-keys"
Oct 30 19:02:29.233974 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.233975    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0062b2f7-32de-4f15-9c6c-7e50fe61584d" containerName="revoke-keys"
Oct 30 19:02:29.234122 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.234109    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0062b2f7-32de-4f15-9c6c-7e50fe61584d" containerName="revoke-keys"
Oct 30 19:02:29.240677 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.240651    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"
Oct 30 19:02:29.295223 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.295184    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"]
Oct 30 19:02:29.295463 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.295428    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:02:29.295632 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.295614    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:02:29.375376 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.375328    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed3861a-54974\" (UID: \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"
Oct 30 19:02:29.375560 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.375441    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed3861a-54974\" (UID: \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"
Oct 30 19:02:29.375608 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.375595    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgpm9\" (UniqueName: \"kubernetes.io/projected/de838a97-cbed-4eca-ac85-26f9ebba6d9a-kube-api-access-jgpm9\") pod \"maas-api-revoke-sub-7ed3861a-54974\" (UID: \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"
Oct 30 19:02:29.477544 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.477502    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgpm9\" (UniqueName: \"kubernetes.io/projected/de838a97-cbed-4eca-ac85-26f9ebba6d9a-kube-api-access-jgpm9\") pod \"maas-api-revoke-sub-7ed3861a-54974\" (UID: \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"
Oct 30 19:02:29.478034 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.478012    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed3861a-54974\" (UID: \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"
Oct 30 19:02:29.478229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.478211    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed3861a-54974\" (UID: \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"
Oct 30 19:02:29.480670 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.480643    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed3861a-54974\" (UID: \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"
Oct 30 19:02:29.482340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.482302    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed3861a-54974\" (UID: \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"
Oct 30 19:02:29.536792 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.536716    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgpm9\" (UniqueName: \"kubernetes.io/projected/de838a97-cbed-4eca-ac85-26f9ebba6d9a-kube-api-access-jgpm9\") pod \"maas-api-revoke-sub-7ed3861a-54974\" (UID: \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"
Oct 30 19:02:29.563541 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.558655    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"
Oct 30 19:02:29.947467 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:29.947428    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"]
Oct 30 19:02:29.947805 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:02:29.947785    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde838a97_cbed_4eca_ac85_26f9ebba6d9a.slice/crio-7743fa1726e9ea347cb3133c60fb9a4af0e55fe434a031f3826a3d164877ac22 WatchSource:0}: Error finding container 7743fa1726e9ea347cb3133c60fb9a4af0e55fe434a031f3826a3d164877ac22: Status 404 returned error can't find the container with id 7743fa1726e9ea347cb3133c60fb9a4af0e55fe434a031f3826a3d164877ac22
Oct 30 19:02:30.090269 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:30.090236    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974" event={"ID":"de838a97-cbed-4eca-ac85-26f9ebba6d9a","Type":"ContainerStarted","Data":"7743fa1726e9ea347cb3133c60fb9a4af0e55fe434a031f3826a3d164877ac22"}
Oct 30 19:02:31.099172 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:31.099063    6014 generic.go:356] "Generic (PLEG): container finished" podID="de838a97-cbed-4eca-ac85-26f9ebba6d9a" containerID="15e038be7f76c396248c41fa1f3138a5690e3735ed929ba536f83f857cf05af1" exitCode=0
Oct 30 19:02:31.099551 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:31.099162    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974" event={"ID":"de838a97-cbed-4eca-ac85-26f9ebba6d9a","Type":"ContainerDied","Data":"15e038be7f76c396248c41fa1f3138a5690e3735ed929ba536f83f857cf05af1"}
Oct 30 19:02:32.308276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:32.308235    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"
Oct 30 19:02:32.426994 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:32.426054    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de838a97-cbed-4eca-ac85-26f9ebba6d9a-kube-api-access-jgpm9\" (UniqueName: \"kubernetes.io/projected/de838a97-cbed-4eca-ac85-26f9ebba6d9a-kube-api-access-jgpm9\") pod \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\" (UID: \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\") "
Oct 30 19:02:32.426994 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:32.426272    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-xks-ca\") pod \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\" (UID: \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\") "
Oct 30 19:02:32.426994 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:32.426360    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-openshift-ca\") pod \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\" (UID: \"de838a97-cbed-4eca-ac85-26f9ebba6d9a\") "
Oct 30 19:02:32.427862 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:32.427831    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-openshift-ca" pod "de838a97-cbed-4eca-ac85-26f9ebba6d9a" (UID: "de838a97-cbed-4eca-ac85-26f9ebba6d9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:32.430227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:32.430197    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-xks-ca" pod "de838a97-cbed-4eca-ac85-26f9ebba6d9a" (UID: "de838a97-cbed-4eca-ac85-26f9ebba6d9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:32.433330 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:32.433301    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de838a97-cbed-4eca-ac85-26f9ebba6d9a-kube-api-access-jgpm9" pod "de838a97-cbed-4eca-ac85-26f9ebba6d9a" (UID: "de838a97-cbed-4eca-ac85-26f9ebba6d9a"). InnerVolumeSpecName "kube-api-access-jgpm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:32.529108 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:32.529073    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:32.529108 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:32.529102    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de838a97-cbed-4eca-ac85-26f9ebba6d9a-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:32.529108 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:32.529113    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgpm9\" (UniqueName: \"kubernetes.io/projected/de838a97-cbed-4eca-ac85-26f9ebba6d9a-kube-api-access-jgpm9\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:33.120236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:33.120204    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"
Oct 30 19:02:33.120236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:33.120215    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974" event={"ID":"de838a97-cbed-4eca-ac85-26f9ebba6d9a","Type":"ContainerDied","Data":"7743fa1726e9ea347cb3133c60fb9a4af0e55fe434a031f3826a3d164877ac22"}
Oct 30 19:02:33.120614 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:33.120300    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7743fa1726e9ea347cb3133c60fb9a4af0e55fe434a031f3826a3d164877ac22"
Oct 30 19:02:36.157062 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.157024    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"]
Oct 30 19:02:36.157888 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.157871    6014 cpu_manager.go:414] "containerMap: removing container" podUID="de838a97-cbed-4eca-ac85-26f9ebba6d9a" containerName="revoke-keys"
Oct 30 19:02:36.157948 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.157890    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de838a97-cbed-4eca-ac85-26f9ebba6d9a" containerName="revoke-keys"
Oct 30 19:02:36.158020 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.158009    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="de838a97-cbed-4eca-ac85-26f9ebba6d9a" containerName="revoke-keys"
Oct 30 19:02:36.161662 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.161644    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.208086 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.208040    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-envoy\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.208299 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.208195    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-credential-socket\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.208376 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.208339    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-data\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.208431 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.208383    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-istio-token\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.208431 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.208411    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-certs\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.208542 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.208515    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwkb2\" (UniqueName: \"kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-kube-api-access-nwkb2\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.208646 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.208627    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681008f7-dc77-453a-a264-fd69fc0fe590-istiod-ca-cert\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.208715 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.208684    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-socket\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.208838 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.208821    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681008f7-dc77-453a-a264-fd69fc0fe590-istio-podinfo\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.236343 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.236316    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cfaf8925-gw-openshift-default-dockercfg-wzt47\""
Oct 30 19:02:36.280981 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.280948    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"]
Oct 30 19:02:36.313210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.311336    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681008f7-dc77-453a-a264-fd69fc0fe590-istiod-ca-cert\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.313210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.311413    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-socket\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.313210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.311522    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681008f7-dc77-453a-a264-fd69fc0fe590-istio-podinfo\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.313210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.311661    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-envoy\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.313210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.311723    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-credential-socket\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.313210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.311901    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-data\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.313210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.311952    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-istio-token\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.313210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.311986    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-certs\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.313210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.312028    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwkb2\" (UniqueName: \"kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-kube-api-access-nwkb2\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.316055 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.315257    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-certs\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.316055 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.315548    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-data\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.316055 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.315722    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681008f7-dc77-453a-a264-fd69fc0fe590-istiod-ca-cert\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.322838 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.322808    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-socket\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.325903 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.323245    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-credential-socket\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.353170 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.348315    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681008f7-dc77-453a-a264-fd69fc0fe590-istio-podinfo\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.361513 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.353688    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-envoy\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.373965 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.373933    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwkb2\" (UniqueName: \"kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-kube-api-access-nwkb2\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.374132 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.374039    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-istio-token\") pod \"e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") " pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.480385 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.480295    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:36.926842 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.926811    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"]
Oct 30 19:02:36.927785 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:02:36.927753    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681008f7_dc77_453a_a264_fd69fc0fe590.slice/crio-2949765fe5e401a44d35deee7c879f6a50a8dddc2a9e0b80adfc7ed41f177a71 WatchSource:0}: Error finding container 2949765fe5e401a44d35deee7c879f6a50a8dddc2a9e0b80adfc7ed41f177a71: Status 404 returned error can't find the container with id 2949765fe5e401a44d35deee7c879f6a50a8dddc2a9e0b80adfc7ed41f177a71
Oct 30 19:02:36.930466 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.930430    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:02:36.930545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.930504    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:02:36.930545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:36.930534    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:02:37.154999 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:37.154936    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s" event={"ID":"681008f7-dc77-453a-a264-fd69fc0fe590","Type":"ContainerStarted","Data":"3fa63dc51d111f3f2668905339eab3bc7321e77a6a96fcbebd806ba5a633e713"}
Oct 30 19:02:37.155266 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:37.155023    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s" event={"ID":"681008f7-dc77-453a-a264-fd69fc0fe590","Type":"ContainerStarted","Data":"2949765fe5e401a44d35deee7c879f6a50a8dddc2a9e0b80adfc7ed41f177a71"}
Oct 30 19:02:37.274391 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:37.274275    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s" podStartSLOduration=1.274253916 podStartE2EDuration="1.274253916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:37.269405523 +0000 UTC m=+1395.912573368" watchObservedRunningTime="2026-10-30 19:02:37.274253916 +0000 UTC m=+1395.917421532"
Oct 30 19:02:37.511390 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:37.511095    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:37.511390 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:37.511246    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:38.163984 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:38.163956    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:38.165000 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:38.164975    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:02:39.240567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.240533    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"]
Oct 30 19:02:39.245245 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.245221    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.350534 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.350504    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-cd57e7dd-openshift-default-dockercfg-9hfh9\""
Oct 30 19:02:39.357888 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.357841    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"]
Oct 30 19:02:39.368874 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.368833    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-envoy\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.369271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.369244    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncgk8\" (UniqueName: \"kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-kube-api-access-ncgk8\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.370452 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.370425    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-certs\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.370724 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.370706    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-token\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.370862 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.370847    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-podinfo\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.371030 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.371015    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-credential-socket\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.371226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.371128    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-socket\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.371382 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.371367    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-data\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.371626 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.371604    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istiod-ca-cert\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.472701 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.472666    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-credential-socket\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.472701 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.472697    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-socket\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.472944 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.472736    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-data\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.472944 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.472785    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istiod-ca-cert\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.473112 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.472844    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-envoy\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.473215 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.473192    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-credential-socket\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.473277 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.473219    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgk8\" (UniqueName: \"kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-kube-api-access-ncgk8\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.473337 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.473310    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-socket\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.473379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.473332    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-data\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.473638 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.473613    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istiod-ca-cert\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.473805 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.473782    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-certs\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.473950 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.473925    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-token\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.474001 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.473986    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-podinfo\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.475477 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.475441    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-certs\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.475822 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.475800    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-envoy\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.483727 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.483695    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-podinfo\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.531333 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.531258    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgk8\" (UniqueName: \"kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-kube-api-access-ncgk8\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.531999 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.531977    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-token\") pod \"e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") " pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:39.564368 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:39.564292    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:40.090864 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:40.090833    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"]
Oct 30 19:02:40.091852 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:02:40.091815    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61093c9_aed4_4f32_85e7_d17180ef7eaf.slice/crio-a2bf847d28f5b2b2de767ced1213defef026bc6eae8343c88fdfccfbfc5bc970 WatchSource:0}: Error finding container a2bf847d28f5b2b2de767ced1213defef026bc6eae8343c88fdfccfbfc5bc970: Status 404 returned error can't find the container with id a2bf847d28f5b2b2de767ced1213defef026bc6eae8343c88fdfccfbfc5bc970
Oct 30 19:02:40.095440 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:40.095324    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:02:40.095440 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:40.095424    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:02:40.095702 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:40.095478    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:02:40.183503 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:40.183471    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw" event={"ID":"b61093c9-aed4-4f32-85e7-d17180ef7eaf","Type":"ContainerStarted","Data":"a2bf847d28f5b2b2de767ced1213defef026bc6eae8343c88fdfccfbfc5bc970"}
Oct 30 19:02:41.193736 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:41.193695    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw" event={"ID":"b61093c9-aed4-4f32-85e7-d17180ef7eaf","Type":"ContainerStarted","Data":"028a9cf53443e6374c8cc5ef56cd7ef5eff5d11b71353449f535d44ccf96d559"}
Oct 30 19:02:41.565368 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:41.565335    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:41.573779 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:41.573610    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:41.693545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:41.693492    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw" podStartSLOduration=2.693478354 podStartE2EDuration="2.693478354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:41.431220671 +0000 UTC m=+1400.074388285" watchObservedRunningTime="2026-10-30 19:02:41.693478354 +0000 UTC m=+1400.336645970"
Oct 30 19:02:42.210462 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:42.210425    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:42.212120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:42.212091    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:02:43.005967 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.005935    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"]
Oct 30 19:02:43.011259 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.011234    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"
Oct 30 19:02:43.063712 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.063679    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:02:43.064021 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.063730    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:02:43.084495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.083205    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlgrk\" (UniqueName: \"kubernetes.io/projected/4427ca03-8e6c-496f-8426-4c5f160307b2-kube-api-access-mlgrk\") pod \"maas-api-revoke-sub-2e3cf824-vnssl\" (UID: \"4427ca03-8e6c-496f-8426-4c5f160307b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"
Oct 30 19:02:43.084495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.083341    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e3cf824-vnssl\" (UID: \"4427ca03-8e6c-496f-8426-4c5f160307b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"
Oct 30 19:02:43.084495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.083383    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e3cf824-vnssl\" (UID: \"4427ca03-8e6c-496f-8426-4c5f160307b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"
Oct 30 19:02:43.116213 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.091798    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"]
Oct 30 19:02:43.185509 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.185464    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlgrk\" (UniqueName: \"kubernetes.io/projected/4427ca03-8e6c-496f-8426-4c5f160307b2-kube-api-access-mlgrk\") pod \"maas-api-revoke-sub-2e3cf824-vnssl\" (UID: \"4427ca03-8e6c-496f-8426-4c5f160307b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"
Oct 30 19:02:43.185763 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.185600    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e3cf824-vnssl\" (UID: \"4427ca03-8e6c-496f-8426-4c5f160307b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"
Oct 30 19:02:43.185763 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.185643    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e3cf824-vnssl\" (UID: \"4427ca03-8e6c-496f-8426-4c5f160307b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"
Oct 30 19:02:43.186750 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.186722    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e3cf824-vnssl\" (UID: \"4427ca03-8e6c-496f-8426-4c5f160307b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"
Oct 30 19:02:43.191903 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.191876    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e3cf824-vnssl\" (UID: \"4427ca03-8e6c-496f-8426-4c5f160307b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"
Oct 30 19:02:43.248166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.244766    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlgrk\" (UniqueName: \"kubernetes.io/projected/4427ca03-8e6c-496f-8426-4c5f160307b2-kube-api-access-mlgrk\") pod \"maas-api-revoke-sub-2e3cf824-vnssl\" (UID: \"4427ca03-8e6c-496f-8426-4c5f160307b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"
Oct 30 19:02:43.330327 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.330289    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"
Oct 30 19:02:43.751639 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:43.751611    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"]
Oct 30 19:02:43.752526 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:02:43.752500    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4427ca03_8e6c_496f_8426_4c5f160307b2.slice/crio-fc8fedfe75c07c40aa18b23e2cbfc9404398e43bdce84b9a1748661a86350323 WatchSource:0}: Error finding container fc8fedfe75c07c40aa18b23e2cbfc9404398e43bdce84b9a1748661a86350323: Status 404 returned error can't find the container with id fc8fedfe75c07c40aa18b23e2cbfc9404398e43bdce84b9a1748661a86350323
Oct 30 19:02:44.235212 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:44.235122    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl" event={"ID":"4427ca03-8e6c-496f-8426-4c5f160307b2","Type":"ContainerStarted","Data":"004d3304ffbc8b80cba400848d4e93d5e22fe38a2737c7f6a4c7cb849dafafb3"}
Oct 30 19:02:44.235447 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:44.235244    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl" event={"ID":"4427ca03-8e6c-496f-8426-4c5f160307b2","Type":"ContainerStarted","Data":"fc8fedfe75c07c40aa18b23e2cbfc9404398e43bdce84b9a1748661a86350323"}
Oct 30 19:02:44.355603 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:44.355550    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl" podStartSLOduration=2.355536562 podStartE2EDuration="2.355536562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:44.351133902 +0000 UTC m=+1402.994301517" watchObservedRunningTime="2026-10-30 19:02:44.355536562 +0000 UTC m=+1402.998704175"
Oct 30 19:02:45.245059 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:45.245027    6014 generic.go:356] "Generic (PLEG): container finished" podID="4427ca03-8e6c-496f-8426-4c5f160307b2" containerID="004d3304ffbc8b80cba400848d4e93d5e22fe38a2737c7f6a4c7cb849dafafb3" exitCode=0
Oct 30 19:02:45.245350 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:45.245106    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl" event={"ID":"4427ca03-8e6c-496f-8426-4c5f160307b2","Type":"ContainerDied","Data":"004d3304ffbc8b80cba400848d4e93d5e22fe38a2737c7f6a4c7cb849dafafb3"}
Oct 30 19:02:46.420224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:46.420198    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"
Oct 30 19:02:46.548590 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:46.548498    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4427ca03-8e6c-496f-8426-4c5f160307b2-kube-api-access-mlgrk\" (UniqueName: \"kubernetes.io/projected/4427ca03-8e6c-496f-8426-4c5f160307b2-kube-api-access-mlgrk\") pod \"4427ca03-8e6c-496f-8426-4c5f160307b2\" (UID: \"4427ca03-8e6c-496f-8426-4c5f160307b2\") "
Oct 30 19:02:46.548800 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:46.548687    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-openshift-ca\") pod \"4427ca03-8e6c-496f-8426-4c5f160307b2\" (UID: \"4427ca03-8e6c-496f-8426-4c5f160307b2\") "
Oct 30 19:02:46.548923 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:46.548905    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-xks-ca\") pod \"4427ca03-8e6c-496f-8426-4c5f160307b2\" (UID: \"4427ca03-8e6c-496f-8426-4c5f160307b2\") "
Oct 30 19:02:46.549074 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:46.549040    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-openshift-ca" pod "4427ca03-8e6c-496f-8426-4c5f160307b2" (UID: "4427ca03-8e6c-496f-8426-4c5f160307b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:46.549819 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:46.549786    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:46.559292 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:46.559261    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-xks-ca" pod "4427ca03-8e6c-496f-8426-4c5f160307b2" (UID: "4427ca03-8e6c-496f-8426-4c5f160307b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:46.559458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:46.559308    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4427ca03-8e6c-496f-8426-4c5f160307b2-kube-api-access-mlgrk" pod "4427ca03-8e6c-496f-8426-4c5f160307b2" (UID: "4427ca03-8e6c-496f-8426-4c5f160307b2"). InnerVolumeSpecName "kube-api-access-mlgrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:46.651581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:46.651542    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4427ca03-8e6c-496f-8426-4c5f160307b2-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:46.651581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:46.651578    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlgrk\" (UniqueName: \"kubernetes.io/projected/4427ca03-8e6c-496f-8426-4c5f160307b2-kube-api-access-mlgrk\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:47.263044 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:47.263006    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl" event={"ID":"4427ca03-8e6c-496f-8426-4c5f160307b2","Type":"ContainerDied","Data":"fc8fedfe75c07c40aa18b23e2cbfc9404398e43bdce84b9a1748661a86350323"}
Oct 30 19:02:47.263044 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:47.263032    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"
Oct 30 19:02:47.263304 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:47.263069    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc8fedfe75c07c40aa18b23e2cbfc9404398e43bdce84b9a1748661a86350323"
Oct 30 19:02:47.859505 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:47.853798    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"]
Oct 30 19:02:47.860295 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:47.860262    6014 cpu_manager.go:414] "containerMap: removing container" podUID="4427ca03-8e6c-496f-8426-4c5f160307b2" containerName="revoke-keys"
Oct 30 19:02:47.860295 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:47.860294    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4427ca03-8e6c-496f-8426-4c5f160307b2" containerName="revoke-keys"
Oct 30 19:02:47.860592 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:47.860538    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="4427ca03-8e6c-496f-8426-4c5f160307b2" containerName="revoke-keys"
Oct 30 19:02:47.873163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:47.866821    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"
Oct 30 19:02:47.923164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:47.920508    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-cd57e7dd-dockercfg-wx56r\""
Oct 30 19:02:47.923164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:47.920599    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-cd57e7dd\""
Oct 30 19:02:47.927201 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:47.923909    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7znj\" (UniqueName: \"kubernetes.io/projected/b976ece9-9e38-4094-8d29-5845669b59a9-kube-api-access-f7znj\") pod \"payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm\" (UID: \"b976ece9-9e38-4094-8d29-5845669b59a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"
Oct 30 19:02:47.927201 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:47.924125    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b976ece9-9e38-4094-8d29-5845669b59a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm\" (UID: \"b976ece9-9e38-4094-8d29-5845669b59a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"
Oct 30 19:02:48.030166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.029467    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7znj\" (UniqueName: \"kubernetes.io/projected/b976ece9-9e38-4094-8d29-5845669b59a9-kube-api-access-f7znj\") pod \"payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm\" (UID: \"b976ece9-9e38-4094-8d29-5845669b59a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"
Oct 30 19:02:48.030166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.029724    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b976ece9-9e38-4094-8d29-5845669b59a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm\" (UID: \"b976ece9-9e38-4094-8d29-5845669b59a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"
Oct 30 19:02:48.066167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.060864    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b976ece9-9e38-4094-8d29-5845669b59a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm\" (UID: \"b976ece9-9e38-4094-8d29-5845669b59a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"
Oct 30 19:02:48.131432 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.118197    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"]
Oct 30 19:02:48.162836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.161835    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7znj\" (UniqueName: \"kubernetes.io/projected/b976ece9-9e38-4094-8d29-5845669b59a9-kube-api-access-f7znj\") pod \"payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm\" (UID: \"b976ece9-9e38-4094-8d29-5845669b59a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"
Oct 30 19:02:48.211220 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.203815    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"
Oct 30 19:02:48.239297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.233912    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"]
Oct 30 19:02:48.240282 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.240253    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"
Oct 30 19:02:48.240396 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.240339    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"]
Oct 30 19:02:48.305831 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.305784    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eeea6a3-87d6-4299-98f9-23cc4f27a912-plugins-config-volume\") pod \"payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4\" (UID: \"3eeea6a3-87d6-4299-98f9-23cc4f27a912\") " pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"
Oct 30 19:02:48.306248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.306222    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pfsl\" (UniqueName: \"kubernetes.io/projected/3eeea6a3-87d6-4299-98f9-23cc4f27a912-kube-api-access-4pfsl\") pod \"payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4\" (UID: \"3eeea6a3-87d6-4299-98f9-23cc4f27a912\") " pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"
Oct 30 19:02:48.408581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.408486    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pfsl\" (UniqueName: \"kubernetes.io/projected/3eeea6a3-87d6-4299-98f9-23cc4f27a912-kube-api-access-4pfsl\") pod \"payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4\" (UID: \"3eeea6a3-87d6-4299-98f9-23cc4f27a912\") " pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"
Oct 30 19:02:48.408729 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.408664    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eeea6a3-87d6-4299-98f9-23cc4f27a912-plugins-config-volume\") pod \"payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4\" (UID: \"3eeea6a3-87d6-4299-98f9-23cc4f27a912\") " pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"
Oct 30 19:02:48.409591 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.409565    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eeea6a3-87d6-4299-98f9-23cc4f27a912-plugins-config-volume\") pod \"payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4\" (UID: \"3eeea6a3-87d6-4299-98f9-23cc4f27a912\") " pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"
Oct 30 19:02:48.466579 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.466540    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pfsl\" (UniqueName: \"kubernetes.io/projected/3eeea6a3-87d6-4299-98f9-23cc4f27a912-kube-api-access-4pfsl\") pod \"payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4\" (UID: \"3eeea6a3-87d6-4299-98f9-23cc4f27a912\") " pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"
Oct 30 19:02:48.621048 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.621019    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"
Oct 30 19:02:48.672191 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:48.672159    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"]
Oct 30 19:02:48.672501 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:02:48.672480    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb976ece9_9e38_4094_8d29_5845669b59a9.slice/crio-0ba64687e56922060e80d60cd06c9c775662544a1c06250e685f4a6f8409af9a WatchSource:0}: Error finding container 0ba64687e56922060e80d60cd06c9c775662544a1c06250e685f4a6f8409af9a: Status 404 returned error can't find the container with id 0ba64687e56922060e80d60cd06c9c775662544a1c06250e685f4a6f8409af9a
Oct 30 19:02:49.004035 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:49.003939    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"]
Oct 30 19:02:49.004562 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:02:49.004423    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eeea6a3_87d6_4299_98f9_23cc4f27a912.slice/crio-5d0d6b07a46728dbb5a4dcd1f8f0e2b262efe27ac4c342af13a4ca07aad9501f WatchSource:0}: Error finding container 5d0d6b07a46728dbb5a4dcd1f8f0e2b262efe27ac4c342af13a4ca07aad9501f: Status 404 returned error can't find the container with id 5d0d6b07a46728dbb5a4dcd1f8f0e2b262efe27ac4c342af13a4ca07aad9501f
Oct 30 19:02:49.303298 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:49.303130    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4" event={"ID":"3eeea6a3-87d6-4299-98f9-23cc4f27a912","Type":"ContainerStarted","Data":"b88805ddf5b09a5dee8aa88ba210d78e33537df78d783bb5eb387bd951860a58"}
Oct 30 19:02:49.303580 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:49.303556    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4" event={"ID":"3eeea6a3-87d6-4299-98f9-23cc4f27a912","Type":"ContainerStarted","Data":"5d0d6b07a46728dbb5a4dcd1f8f0e2b262efe27ac4c342af13a4ca07aad9501f"}
Oct 30 19:02:49.303718 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:49.303603    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"
Oct 30 19:02:49.305432 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:49.305405    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm" event={"ID":"b976ece9-9e38-4094-8d29-5845669b59a9","Type":"ContainerStarted","Data":"b17b6265c84170422b58de8a9ddeec7c4dfbf2dd341d821012ba5963fa30d487"}
Oct 30 19:02:49.305610 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:49.305453    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm" event={"ID":"b976ece9-9e38-4094-8d29-5845669b59a9","Type":"ContainerStarted","Data":"0ba64687e56922060e80d60cd06c9c775662544a1c06250e685f4a6f8409af9a"}
Oct 30 19:02:49.305610 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:49.305565    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"
Oct 30 19:02:49.417532 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:49.417470    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4" podStartSLOduration=2.417452622 podStartE2EDuration="2.417452622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:49.415326059 +0000 UTC m=+1408.058493674" watchObservedRunningTime="2026-10-30 19:02:49.417452622 +0000 UTC m=+1408.060620235"
Oct 30 19:02:49.533615 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:49.533551    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm" podStartSLOduration=2.533531967 podStartE2EDuration="2.533531967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:49.529028646 +0000 UTC m=+1408.172196287" watchObservedRunningTime="2026-10-30 19:02:49.533531967 +0000 UTC m=+1408.176699581"
Oct 30 19:02:53.261456 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:02:53.261414    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:02:55.086867 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:55.086827    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:02:56.104173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:56.087903    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"]
Oct 30 19:02:56.134165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:56.132523    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-364e71a8-9vbsj"]
Oct 30 19:02:57.560166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:57.557383    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5b6fbb-a1b8-43e6-b847-dd6102e1b603" path="/var/lib/kubelet/pods/fd5b6fbb-a1b8-43e6-b847-dd6102e1b603/volumes"
Oct 30 19:02:57.692286 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:57.692231    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:57.895501 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:57.895432    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": context deadline exceeded"
Oct 30 19:02:57.991297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:57.991246    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.132177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:58.130436    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.132177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:58.130699    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:02:58.237249 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:58.222297    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.330187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:58.329235    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.440050 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:58.440008    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.564241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:58.557790    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.710322 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:58.710255    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:02:58.711214 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:58.711191    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:02:58.917235 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:58.896279    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.917235 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:58.897069    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:02:59.059258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.042355    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Oct 30 19:02:59.059258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.043041    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:02:59.108169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.081812    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:02:59.224221 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.181757    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:02:59.226396 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.226352    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:02:59.264493 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.264461    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 19:02:59.407168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.355049    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.407168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.386519    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:02:59.448749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.407714    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:02:59.448749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.446369    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.572225 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.570759    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.581317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.581288    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:02:59.716166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.703496    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:02:59.794165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.790231    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" podUID="898fcd26-1c1e-40e6-994d-619340f782cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.827403 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.716434    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.920164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.914910    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.945167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:02:59.932804    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.054287 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.052382    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:03:00.217444 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.217337    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:03:00.287332 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.287282    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.287837 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.287803    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:03:00.324992 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.324948    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:03:00.341564 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.341521    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.366375 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.363923    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.564224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.563353    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:03:00.564224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.563415    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm" podUID="5278d58c-cb56-4847-88d1-e366fad02e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.672324 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.672270    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.745823 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.728459    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:03:00.763276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.763244    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"
Oct 30 19:03:00.819228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.790439    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"
Oct 30 19:03:00.904176 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.858230    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.944267 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.944218    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:01.008632 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:00.945162    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:01.180228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:01.064182    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:03:01.696169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:01.673180    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:01.869445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:01.681496    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:03:02.018234 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:01.681684    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:03:02.018234 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:01.780799    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c" podUID="5596ab6c-2bae-460e-81c1-e875ce418e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:03:02.044559 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:01.796496    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:03:02.114108 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.108680    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:02.119720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.119230    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:02.236503 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.113733    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:02.359643 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.349715    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:02.464561 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.464530    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:03:02.464806 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.464792    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"]
Oct 30 19:03:02.475842 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.474666    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"]
Oct 30 19:03:02.475842 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.474804    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:03:02.475842 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.474841    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 19:03:02.475842 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.474876    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:03:02.475842 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.474933    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:03:02.476259 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.476028    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"
Oct 30 19:03:02.478867 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.478844    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:03:02.479161 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.479122    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:03:02.481217 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.481193    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:03:02.531537 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.531506    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:03:02.531719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.531606    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:03:02.545283 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.545246    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6cc08ed-6dct9\" (UID: \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"
Oct 30 19:03:02.546709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.545964    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxnls\" (UniqueName: \"kubernetes.io/projected/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-kube-api-access-jxnls\") pod \"maas-api-revoke-sub-a6cc08ed-6dct9\" (UID: \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"
Oct 30 19:03:02.546709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.546112    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6cc08ed-6dct9\" (UID: \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"
Oct 30 19:03:02.648538 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.648441    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxnls\" (UniqueName: \"kubernetes.io/projected/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-kube-api-access-jxnls\") pod \"maas-api-revoke-sub-a6cc08ed-6dct9\" (UID: \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"
Oct 30 19:03:02.648538 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.648524    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6cc08ed-6dct9\" (UID: \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"
Oct 30 19:03:02.648784 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.648748    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6cc08ed-6dct9\" (UID: \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"
Oct 30 19:03:02.649405 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.649383    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6cc08ed-6dct9\" (UID: \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"
Oct 30 19:03:02.651260 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.651234    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6cc08ed-6dct9\" (UID: \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"
Oct 30 19:03:02.710789 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.710758    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxnls\" (UniqueName: \"kubernetes.io/projected/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-kube-api-access-jxnls\") pod \"maas-api-revoke-sub-a6cc08ed-6dct9\" (UID: \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"
Oct 30 19:03:02.799913 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:02.799881    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"
Oct 30 19:03:03.193014 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:03:03.192979    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f86f50_d5a0_4288_bc8f_0c45ea7fb845.slice/crio-cac1d54beb673dd929268cf57b6a3a27dafaf718b61976ec540de7b3b72a7582 WatchSource:0}: Error finding container cac1d54beb673dd929268cf57b6a3a27dafaf718b61976ec540de7b3b72a7582: Status 404 returned error can't find the container with id cac1d54beb673dd929268cf57b6a3a27dafaf718b61976ec540de7b3b72a7582
Oct 30 19:03:03.194636 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:03.194604    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"]
Oct 30 19:03:03.412228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:03.412194    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9" event={"ID":"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845","Type":"ContainerStarted","Data":"cac1d54beb673dd929268cf57b6a3a27dafaf718b61976ec540de7b3b72a7582"}
Oct 30 19:03:04.421634 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:04.421553    6014 generic.go:356] "Generic (PLEG): container finished" podID="e1f86f50-d5a0-4288-bc8f-0c45ea7fb845" containerID="23c7315790873256077b4d593263230fdbbaddea0a0b85b3303d347d8ddbae3a" exitCode=0
Oct 30 19:03:04.422014 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:04.421637    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9" event={"ID":"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845","Type":"ContainerDied","Data":"23c7315790873256077b4d593263230fdbbaddea0a0b85b3303d347d8ddbae3a"}
Oct 30 19:03:05.448587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.448555    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"]
Oct 30 19:03:05.456459 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.456431    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.508715 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.508683    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:03:05.508878 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.508769    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:03:05.508989 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.508896    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:03:05.509067 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.509050    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5sqr9\""
Oct 30 19:03:05.563654 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.563619    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"]
Oct 30 19:03:05.615775 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.615740    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.615961 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.615792    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.615961 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.615903    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.616084 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.616010    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.616084 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.616046    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.616258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.616227    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.616313 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.616273    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltn5z\" (UniqueName: \"kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-kube-api-access-ltn5z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.622189 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.622160    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"
Oct 30 19:03:05.718003 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.717917    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-xks-ca\") pod \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\" (UID: \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\") "
Oct 30 19:03:05.718003 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.717971    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-openshift-ca\") pod \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\" (UID: \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\") "
Oct 30 19:03:05.718315 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.718206    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-kube-api-access-jxnls\" (UniqueName: \"kubernetes.io/projected/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-kube-api-access-jxnls\") pod \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\" (UID: \"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845\") "
Oct 30 19:03:05.718454 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.718431    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-openshift-ca" pod "e1f86f50-d5a0-4288-bc8f-0c45ea7fb845" (UID: "e1f86f50-d5a0-4288-bc8f-0c45ea7fb845"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:05.718603 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.718580    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.718767 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.718621    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.718767 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.718739    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.718767 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.718781    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltn5z\" (UniqueName: \"kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-kube-api-access-ltn5z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.719064 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.718904    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.719064 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.718955    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.719064 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.719032    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.719336 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.719176    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:03:05.719468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.719439    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.721688 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.721663    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-xks-ca" pod "e1f86f50-d5a0-4288-bc8f-0c45ea7fb845" (UID: "e1f86f50-d5a0-4288-bc8f-0c45ea7fb845"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:05.721781 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.721719    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-kube-api-access-jxnls" pod "e1f86f50-d5a0-4288-bc8f-0c45ea7fb845" (UID: "e1f86f50-d5a0-4288-bc8f-0c45ea7fb845"). InnerVolumeSpecName "kube-api-access-jxnls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:05.721942 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.721921    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.722496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.722475    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.722786 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.722764    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.722911 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.722895    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.781545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.781451    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltn5z\" (UniqueName: \"kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-kube-api-access-ltn5z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.781545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.781494    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bgxf7\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:05.821011 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.820976    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxnls\" (UniqueName: \"kubernetes.io/projected/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-kube-api-access-jxnls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:03:05.821011 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:05.821005    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:03:06.073709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:06.073676    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:06.442861 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:06.442776    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"
Oct 30 19:03:06.442861 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:06.442802    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9" event={"ID":"e1f86f50-d5a0-4288-bc8f-0c45ea7fb845","Type":"ContainerDied","Data":"cac1d54beb673dd929268cf57b6a3a27dafaf718b61976ec540de7b3b72a7582"}
Oct 30 19:03:06.443679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:06.442872    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cac1d54beb673dd929268cf57b6a3a27dafaf718b61976ec540de7b3b72a7582"
Oct 30 19:03:06.513737 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:03:06.513707    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a26ab8b_b868_4e14_bf8b_6ba8cf4a3279.slice/crio-5baa65bc454555369ec55171ed440616da88e731731083588b789257dc1c769d WatchSource:0}: Error finding container 5baa65bc454555369ec55171ed440616da88e731731083588b789257dc1c769d: Status 404 returned error can't find the container with id 5baa65bc454555369ec55171ed440616da88e731731083588b789257dc1c769d
Oct 30 19:03:06.516544 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:06.516523    6014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:03:06.576422 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:06.576380    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"]
Oct 30 19:03:07.452678 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:07.452638    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7" event={"ID":"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279","Type":"ContainerStarted","Data":"5baa65bc454555369ec55171ed440616da88e731731083588b789257dc1c769d"}
Oct 30 19:03:10.095013 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:10.094977    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:03:10.095376 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:10.095044    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:03:10.491391 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:10.491281    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7" event={"ID":"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279","Type":"ContainerStarted","Data":"aeed9f033663a4831e9c121022c3946ce5fdada900889bf1469d894786e4ee6c"}
Oct 30 19:03:10.491688 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:10.491637    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:10.707731 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:10.707675    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7" podStartSLOduration=2.130664089 podStartE2EDuration="5.707659692s" totalImagesPullingTime="3.576995603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:10.655273701 +0000 UTC m=+1429.298441315" watchObservedRunningTime="2026-10-30 19:03:10.707659692 +0000 UTC m=+1429.350827306"
Oct 30 19:03:11.509770 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:11.509731    6014 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bgxf7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:03:11.510314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:11.509795    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7" podUID="6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:03:14.504293 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:14.504264    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:03:21.172010 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.171972    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"]
Oct 30 19:03:21.172830 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.172813    6014 cpu_manager.go:414] "containerMap: removing container" podUID="e1f86f50-d5a0-4288-bc8f-0c45ea7fb845" containerName="revoke-keys"
Oct 30 19:03:21.172875 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.172831    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f86f50-d5a0-4288-bc8f-0c45ea7fb845" containerName="revoke-keys"
Oct 30 19:03:21.172976 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.172966    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f86f50-d5a0-4288-bc8f-0c45ea7fb845" containerName="revoke-keys"
Oct 30 19:03:21.178023 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.178002    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.289321 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.289278    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.289502 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.289347    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.289552 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.289503    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.289651 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.289632    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9723156-3755-4457-8694-8ef4a234d191-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.289714 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.289697    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmsg8\" (UniqueName: \"kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-kube-api-access-rmsg8\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.289756 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.289731    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.289872 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.289857    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.290013 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.289991    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.290113 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.290095    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9723156-3755-4457-8694-8ef4a234d191-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.381270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.381232    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"]
Oct 30 19:03:21.391814 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.391778    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.391997 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.391829    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.391997 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.391870    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.391997 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.391931    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9723156-3755-4457-8694-8ef4a234d191-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.391997 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.391986    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmsg8\" (UniqueName: \"kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-kube-api-access-rmsg8\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.392253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.392018    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.392253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.392107    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.392253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.392229    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.392411 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.392306    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9723156-3755-4457-8694-8ef4a234d191-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.392473 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.392438    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.393050 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.392733    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.393050 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.392755    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.393050 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.392809    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.393669 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.393648    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9723156-3755-4457-8694-8ef4a234d191-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.395625 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.395606    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.395929 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.395907    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9723156-3755-4457-8694-8ef4a234d191-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.448412 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.448345    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.448716 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.448531    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmsg8\" (UniqueName: \"kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-kube-api-access-rmsg8\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.494535 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.494493    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:21.928527 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:03:21.928497    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9723156_3755_4457_8694_8ef4a234d191.slice/crio-5ce1494968f5dfe2bb50e52d92398f2048066608f13a79570286b778ef598f7b WatchSource:0}: Error finding container 5ce1494968f5dfe2bb50e52d92398f2048066608f13a79570286b778ef598f7b: Status 404 returned error can't find the container with id 5ce1494968f5dfe2bb50e52d92398f2048066608f13a79570286b778ef598f7b
Oct 30 19:03:21.928933 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.928911    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"]
Oct 30 19:03:21.931391 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.931359    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:03:21.931514 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.931434    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:03:21.931514 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:21.931487    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:03:22.619891 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:22.619852    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" event={"ID":"c9723156-3755-4457-8694-8ef4a234d191","Type":"ContainerStarted","Data":"49057df9c5d6601da560d86fa625a273e9062bd6ae9036b6d7f59cb022340e80"}
Oct 30 19:03:22.620518 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:22.619920    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" event={"ID":"c9723156-3755-4457-8694-8ef4a234d191","Type":"ContainerStarted","Data":"5ce1494968f5dfe2bb50e52d92398f2048066608f13a79570286b778ef598f7b"}
Oct 30 19:03:22.788396 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:22.788335    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" podStartSLOduration=1.788320997 podStartE2EDuration="1.788320997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:22.786372528 +0000 UTC m=+1441.429540144" watchObservedRunningTime="2026-10-30 19:03:22.788320997 +0000 UTC m=+1441.431488683"
Oct 30 19:03:23.284723 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:03:23.284684    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:03:23.426035 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:23.425997    6014 scope.go:122] "RemoveContainer" containerID="458b4bec26fa9173721518cd3d02b0385da3b1b36f40826aa5ccb5f5c2400bda"
Oct 30 19:03:23.437380 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:23.437357    6014 scope.go:122] "RemoveContainer" containerID="ec16f98a3c3bca299d42f69a7de99960fe5e60f13d1e5246ffbea128d8346c7b"
Oct 30 19:03:23.463983 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:23.462578    6014 scope.go:122] "RemoveContainer" containerID="02b2137b89c111b507a82ae452a012d123a7d2d95dcdf95c777f40e90c383971"
Oct 30 19:03:23.494753 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:23.494727    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:23.496250 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:23.496205    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:03:23.496335 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:23.496253    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" podUID="c9723156-3755-4457-8694-8ef4a234d191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:03:24.494919 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:24.494869    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:03:24.495376 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:24.494940    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" podUID="c9723156-3755-4457-8694-8ef4a234d191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:03:25.495080 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:25.495049    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:03:25.495607 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:25.495105    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" podUID="c9723156-3755-4457-8694-8ef4a234d191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:03:26.495005 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:26.494966    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:03:26.495248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:26.495028    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" podUID="c9723156-3755-4457-8694-8ef4a234d191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:03:27.494824 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:27.494791    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:03:27.495016 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:27.494844    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" podUID="c9723156-3755-4457-8694-8ef4a234d191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:03:28.494910 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:28.494868    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:03:28.495319 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:28.494954    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" podUID="c9723156-3755-4457-8694-8ef4a234d191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:03:30.095164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:30.093496    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:30.095887 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:30.095608    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:30.121495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:30.121457    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:03:30.975165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:30.974959    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:31.683165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:31.679396    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:31.907166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:31.906994    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:31.907392 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:31.907324    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:03:51.188552 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:51.188509    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:03:53.552251 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:03:53.551589    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:03:53.866232 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:53.854230    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:53.876897 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:53.871673    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:53.917169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:53.917046    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:53.991210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:53.978231    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:54.453167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.449292    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:54.453167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.452402    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"]
Oct 30 19:03:54.464174 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.463539    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"
Oct 30 19:03:54.665168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.649374    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:54.665168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.659495    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:03:54.665168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.662254    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:03:54.690163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.681486    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:54.742417 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.722392    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hxk5\" (UniqueName: \"kubernetes.io/projected/307904db-214a-46d3-9434-b7ece8f648c5-kube-api-access-7hxk5\") pod \"maas-api-revoke-sub-ad6cf069-w9lgb\" (UID: \"307904db-214a-46d3-9434-b7ece8f648c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"
Oct 30 19:03:54.910500 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.891282    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad6cf069-w9lgb\" (UID: \"307904db-214a-46d3-9434-b7ece8f648c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"
Oct 30 19:03:54.910500 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.891402    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad6cf069-w9lgb\" (UID: \"307904db-214a-46d3-9434-b7ece8f648c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"
Oct 30 19:03:54.910500 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.768510    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"]
Oct 30 19:03:54.910500 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.890495    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:54.910500 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.890299    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:03:54.910500 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.890442    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:54.918179 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.912123    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:03:54.918179 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.912243    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:03:54.927845 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.918890    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": context deadline exceeded"
Oct 30 19:03:54.940375 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:54.929162    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:03:55.045469 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.045368    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:03:55.058263 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.058227    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:03:55.061103 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.061063    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:03:55.186501 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.186120    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hxk5\" (UniqueName: \"kubernetes.io/projected/307904db-214a-46d3-9434-b7ece8f648c5-kube-api-access-7hxk5\") pod \"maas-api-revoke-sub-ad6cf069-w9lgb\" (UID: \"307904db-214a-46d3-9434-b7ece8f648c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"
Oct 30 19:03:55.186501 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.186236    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad6cf069-w9lgb\" (UID: \"307904db-214a-46d3-9434-b7ece8f648c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"
Oct 30 19:03:55.196686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.196637    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad6cf069-w9lgb\" (UID: \"307904db-214a-46d3-9434-b7ece8f648c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"
Oct 30 19:03:55.238531 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.221346    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad6cf069-w9lgb\" (UID: \"307904db-214a-46d3-9434-b7ece8f648c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"
Oct 30 19:03:55.292332 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.292285    6014 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:55.327240 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.318611    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad6cf069-w9lgb\" (UID: \"307904db-214a-46d3-9434-b7ece8f648c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"
Oct 30 19:03:55.327240 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.322834    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:03:55.597255 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.593014    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hxk5\" (UniqueName: \"kubernetes.io/projected/307904db-214a-46d3-9434-b7ece8f648c5-kube-api-access-7hxk5\") pod \"maas-api-revoke-sub-ad6cf069-w9lgb\" (UID: \"307904db-214a-46d3-9434-b7ece8f648c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"
Oct 30 19:03:55.726034 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.726005    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:03:55.726714 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.726689    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:03:55.726836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.726740    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:03:55.729158 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:55.729116    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"
Oct 30 19:03:56.252168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:56.247534    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"]
Oct 30 19:03:56.729472 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:56.729440    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb" event={"ID":"307904db-214a-46d3-9434-b7ece8f648c5","Type":"ContainerStarted","Data":"b630c3da57f0f9713275835b5883ec2dcecc2729153485174065a64b714c6f77"}
Oct 30 19:03:57.739718 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:57.739693    6014 generic.go:356] "Generic (PLEG): container finished" podID="307904db-214a-46d3-9434-b7ece8f648c5" containerID="578d9a07b4958e33045900f03913d935967681b157323f709cc8a39dac59a6b7" exitCode=0
Oct 30 19:03:57.740032 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:57.739881    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb" event={"ID":"307904db-214a-46d3-9434-b7ece8f648c5","Type":"ContainerDied","Data":"578d9a07b4958e33045900f03913d935967681b157323f709cc8a39dac59a6b7"}
Oct 30 19:03:59.001791 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:59.001725    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"
Oct 30 19:03:59.121212 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:59.121172    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-xks-ca\") pod \"307904db-214a-46d3-9434-b7ece8f648c5\" (UID: \"307904db-214a-46d3-9434-b7ece8f648c5\") "
Oct 30 19:03:59.121558 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:59.121531    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-openshift-ca\") pod \"307904db-214a-46d3-9434-b7ece8f648c5\" (UID: \"307904db-214a-46d3-9434-b7ece8f648c5\") "
Oct 30 19:03:59.122005 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:59.121986    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307904db-214a-46d3-9434-b7ece8f648c5-kube-api-access-7hxk5\" (UniqueName: \"kubernetes.io/projected/307904db-214a-46d3-9434-b7ece8f648c5-kube-api-access-7hxk5\") pod \"307904db-214a-46d3-9434-b7ece8f648c5\" (UID: \"307904db-214a-46d3-9434-b7ece8f648c5\") "
Oct 30 19:03:59.124517 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:59.124492    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-openshift-ca" pod "307904db-214a-46d3-9434-b7ece8f648c5" (UID: "307904db-214a-46d3-9434-b7ece8f648c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:59.186344 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:59.186299    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307904db-214a-46d3-9434-b7ece8f648c5-kube-api-access-7hxk5" pod "307904db-214a-46d3-9434-b7ece8f648c5" (UID: "307904db-214a-46d3-9434-b7ece8f648c5"). InnerVolumeSpecName "kube-api-access-7hxk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:59.202968 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:59.202911    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-xks-ca" pod "307904db-214a-46d3-9434-b7ece8f648c5" (UID: "307904db-214a-46d3-9434-b7ece8f648c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:59.226165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:59.226112    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hxk5\" (UniqueName: \"kubernetes.io/projected/307904db-214a-46d3-9434-b7ece8f648c5-kube-api-access-7hxk5\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:03:59.226355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:59.226190    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:03:59.226355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:59.226208    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307904db-214a-46d3-9434-b7ece8f648c5-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:03:59.762744 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:59.762647    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb" event={"ID":"307904db-214a-46d3-9434-b7ece8f648c5","Type":"ContainerDied","Data":"b630c3da57f0f9713275835b5883ec2dcecc2729153485174065a64b714c6f77"}
Oct 30 19:03:59.762744 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:59.762703    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b630c3da57f0f9713275835b5883ec2dcecc2729153485174065a64b714c6f77"
Oct 30 19:03:59.762744 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:03:59.762655    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"
Oct 30 19:04:03.810925 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:03.810835    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s" event={"ID":"681008f7-dc77-453a-a264-fd69fc0fe590","Type":"ContainerDied","Data":"3fa63dc51d111f3f2668905339eab3bc7321e77a6a96fcbebd806ba5a633e713"}
Oct 30 19:04:03.823879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:03.810751    6014 generic.go:356] "Generic (PLEG): container finished" podID="681008f7-dc77-453a-a264-fd69fc0fe590" containerID="3fa63dc51d111f3f2668905339eab3bc7321e77a6a96fcbebd806ba5a633e713" exitCode=0
Oct 30 19:04:03.823879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:03.811533    6014 scope.go:122] "RemoveContainer" containerID="3fa63dc51d111f3f2668905339eab3bc7321e77a6a96fcbebd806ba5a633e713"
Oct 30 19:04:03.823879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:03.813655    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:04:03.823879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:03.813727    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:04:03.823879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:03.813763    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:04:04.820539 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:04.820506    6014 generic.go:356] "Generic (PLEG): container finished" podID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerID="028a9cf53443e6374c8cc5ef56cd7ef5eff5d11b71353449f535d44ccf96d559" exitCode=0
Oct 30 19:04:04.820960 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:04.820582    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw" event={"ID":"b61093c9-aed4-4f32-85e7-d17180ef7eaf","Type":"ContainerDied","Data":"028a9cf53443e6374c8cc5ef56cd7ef5eff5d11b71353449f535d44ccf96d559"}
Oct 30 19:04:04.821117 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:04.821098    6014 scope.go:122] "RemoveContainer" containerID="028a9cf53443e6374c8cc5ef56cd7ef5eff5d11b71353449f535d44ccf96d559"
Oct 30 19:04:04.822924 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:04.822900    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s" event={"ID":"681008f7-dc77-453a-a264-fd69fc0fe590","Type":"ContainerStarted","Data":"b5c714fd8f7ea617153832fd1428f8d3237dc0b83e8f094ea2500ab8379c431d"}
Oct 30 19:04:04.823019 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:04.822916    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:04:04.823019 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:04.822985    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:04:04.823019 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:04.823016    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:04:05.482343 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:05.482311    6014 patch_prober.go:29] interesting pod/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:04:05.482552 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:05.482371    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s" podUID="681008f7-dc77-453a-a264-fd69fc0fe590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 19:04:05.486493 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:05.486463    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:04:05.838352 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:05.838315    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw" event={"ID":"b61093c9-aed4-4f32-85e7-d17180ef7eaf","Type":"ContainerStarted","Data":"bd810839730995db5de0fd221120a5bd4719528a44e150ecd3d7520d89b70480"}
Oct 30 19:04:06.491170 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:06.490648    6014 patch_prober.go:29] interesting pod/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:04:06.491170 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:06.490714    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s" podUID="681008f7-dc77-453a-a264-fd69fc0fe590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 19:04:06.569788 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:06.569097    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:04:06.582596 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:06.582563    6014 patch_prober.go:29] interesting pod/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:04:06.582898 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:06.582869    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw" podUID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 19:04:07.497837 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:07.497802    6014 patch_prober.go:29] interesting pod/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:04:07.498528 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:07.498479    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s" podUID="681008f7-dc77-453a-a264-fd69fc0fe590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 19:04:07.569559 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:07.569525    6014 patch_prober.go:29] interesting pod/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:04:07.569874 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:07.569842    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw" podUID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 19:04:08.165167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:08.164246    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:04:08.481661 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:08.481564    6014 patch_prober.go:29] interesting pod/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:04:08.481978 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:08.481947    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s" podUID="681008f7-dc77-453a-a264-fd69fc0fe590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 19:04:08.567744 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:08.567697    6014 patch_prober.go:29] interesting pod/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:04:08.568410 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:08.568360    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw" podUID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 19:04:09.486608 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:09.486567    6014 patch_prober.go:29] interesting pod/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:04:09.486930 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:09.486903    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s" podUID="681008f7-dc77-453a-a264-fd69fc0fe590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 19:04:09.570921 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:09.570888    6014 patch_prober.go:29] interesting pod/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:04:09.571553 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:09.571525    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw" podUID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 19:04:10.238828 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.238343    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"]
Oct 30 19:04:10.243182 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.242914    6014 cpu_manager.go:414] "containerMap: removing container" podUID="307904db-214a-46d3-9434-b7ece8f648c5" containerName="revoke-keys"
Oct 30 19:04:10.243182 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.242948    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307904db-214a-46d3-9434-b7ece8f648c5" containerName="revoke-keys"
Oct 30 19:04:10.243519 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.243494    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="307904db-214a-46d3-9434-b7ece8f648c5" containerName="revoke-keys"
Oct 30 19:04:10.259542 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.259509    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"
Oct 30 19:04:10.325422 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.325387    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"]
Oct 30 19:04:10.346998 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.346962    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6355210-9wbqk\" (UID: \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"
Oct 30 19:04:10.347294 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.347028    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctjjs\" (UniqueName: \"kubernetes.io/projected/cb4641dc-c870-4d9d-aee0-265967e8c5d3-kube-api-access-ctjjs\") pod \"maas-api-revoke-sub-d6355210-9wbqk\" (UID: \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"
Oct 30 19:04:10.347294 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.347081    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6355210-9wbqk\" (UID: \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"
Oct 30 19:04:10.384127 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.384094    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:04:10.384391 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.384363    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:04:10.463163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.454472    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6355210-9wbqk\" (UID: \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"
Oct 30 19:04:10.463163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.454569    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctjjs\" (UniqueName: \"kubernetes.io/projected/cb4641dc-c870-4d9d-aee0-265967e8c5d3-kube-api-access-ctjjs\") pod \"maas-api-revoke-sub-d6355210-9wbqk\" (UID: \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"
Oct 30 19:04:10.463163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.454659    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6355210-9wbqk\" (UID: \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"
Oct 30 19:04:10.463163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.460105    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6355210-9wbqk\" (UID: \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"
Oct 30 19:04:10.497446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.497364    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6355210-9wbqk\" (UID: \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"
Oct 30 19:04:10.517762 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.517724    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctjjs\" (UniqueName: \"kubernetes.io/projected/cb4641dc-c870-4d9d-aee0-265967e8c5d3-kube-api-access-ctjjs\") pod \"maas-api-revoke-sub-d6355210-9wbqk\" (UID: \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"
Oct 30 19:04:10.527202 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.526067    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:04:10.528621 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.528597    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:04:10.570713 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.570676    6014 patch_prober.go:29] interesting pod/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:04:10.571011 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.570981    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw" podUID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 19:04:10.620715 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:10.620682    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"
Oct 30 19:04:11.047205 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:11.047107    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"]
Oct 30 19:04:11.570015 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:11.569983    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:04:11.570475 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:11.570455    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:04:11.571215 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:11.571196    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:04:11.952830 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:11.952796    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk" event={"ID":"cb4641dc-c870-4d9d-aee0-265967e8c5d3","Type":"ContainerStarted","Data":"13a46748040eb62f7d3b248f10867c65d0a46ae144f1f838077679aa272dbaa7"}
Oct 30 19:04:11.953052 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:11.952859    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk" event={"ID":"cb4641dc-c870-4d9d-aee0-265967e8c5d3","Type":"ContainerStarted","Data":"35dbe8314918246381a1640145a1d791265de8ca2af950abb44679f106e4b336"}
Oct 30 19:04:12.067808 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:12.067755    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk" podStartSLOduration=2.067739735 podStartE2EDuration="2.067739735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:12.06444487 +0000 UTC m=+1490.707612485" watchObservedRunningTime="2026-10-30 19:04:12.067739735 +0000 UTC m=+1490.710907348"
Oct 30 19:04:12.962554 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:12.962515    6014 generic.go:356] "Generic (PLEG): container finished" podID="cb4641dc-c870-4d9d-aee0-265967e8c5d3" containerID="13a46748040eb62f7d3b248f10867c65d0a46ae144f1f838077679aa272dbaa7" exitCode=0
Oct 30 19:04:12.962963 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:12.962599    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk" event={"ID":"cb4641dc-c870-4d9d-aee0-265967e8c5d3","Type":"ContainerDied","Data":"13a46748040eb62f7d3b248f10867c65d0a46ae144f1f838077679aa272dbaa7"}
Oct 30 19:04:14.158801 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:14.158769    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"
Oct 30 19:04:14.269665 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:14.269568    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb4641dc-c870-4d9d-aee0-265967e8c5d3-kube-api-access-ctjjs\" (UniqueName: \"kubernetes.io/projected/cb4641dc-c870-4d9d-aee0-265967e8c5d3-kube-api-access-ctjjs\") pod \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\" (UID: \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\") "
Oct 30 19:04:14.269859 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:14.269831    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-xks-ca\") pod \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\" (UID: \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\") "
Oct 30 19:04:14.269964 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:14.269935    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-openshift-ca\") pod \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\" (UID: \"cb4641dc-c870-4d9d-aee0-265967e8c5d3\") "
Oct 30 19:04:14.270370 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:14.270341    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-openshift-ca" pod "cb4641dc-c870-4d9d-aee0-265967e8c5d3" (UID: "cb4641dc-c870-4d9d-aee0-265967e8c5d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:14.270976 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:14.270951    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:14.272887 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:14.272845    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4641dc-c870-4d9d-aee0-265967e8c5d3-kube-api-access-ctjjs" pod "cb4641dc-c870-4d9d-aee0-265967e8c5d3" (UID: "cb4641dc-c870-4d9d-aee0-265967e8c5d3"). InnerVolumeSpecName "kube-api-access-ctjjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:14.274303 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:14.274276    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-xks-ca" pod "cb4641dc-c870-4d9d-aee0-265967e8c5d3" (UID: "cb4641dc-c870-4d9d-aee0-265967e8c5d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:14.372112 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:14.372073    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb4641dc-c870-4d9d-aee0-265967e8c5d3-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:14.372112 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:14.372105    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctjjs\" (UniqueName: \"kubernetes.io/projected/cb4641dc-c870-4d9d-aee0-265967e8c5d3-kube-api-access-ctjjs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:14.984853 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:14.984825    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"
Oct 30 19:04:14.985039 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:14.984829    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk" event={"ID":"cb4641dc-c870-4d9d-aee0-265967e8c5d3","Type":"ContainerDied","Data":"35dbe8314918246381a1640145a1d791265de8ca2af950abb44679f106e4b336"}
Oct 30 19:04:14.985039 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:14.984948    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35dbe8314918246381a1640145a1d791265de8ca2af950abb44679f106e4b336"
Oct 30 19:04:21.857012 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:21.856979    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"]
Oct 30 19:04:21.857816 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:21.857801    6014 cpu_manager.go:414] "containerMap: removing container" podUID="cb4641dc-c870-4d9d-aee0-265967e8c5d3" containerName="revoke-keys"
Oct 30 19:04:21.857861 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:21.857818    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb4641dc-c870-4d9d-aee0-265967e8c5d3" containerName="revoke-keys"
Oct 30 19:04:21.857971 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:21.857961    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb4641dc-c870-4d9d-aee0-265967e8c5d3" containerName="revoke-keys"
Oct 30 19:04:21.861222 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:21.861205    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"
Oct 30 19:04:21.919956 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:21.919930    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"]
Oct 30 19:04:21.962492 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:21.962456    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:04:21.962640 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:21.962557    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:04:22.010068 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:22.010022    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvklf\" (UniqueName: \"kubernetes.io/projected/2f7c308e-730e-4a27-b3c5-3a39071c66a3-kube-api-access-wvklf\") pod \"maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6\" (UID: \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"
Oct 30 19:04:22.010334 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:22.010316    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6\" (UID: \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"
Oct 30 19:04:22.010494 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:22.010477    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6\" (UID: \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"
Oct 30 19:04:22.114940 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:22.114822    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvklf\" (UniqueName: \"kubernetes.io/projected/2f7c308e-730e-4a27-b3c5-3a39071c66a3-kube-api-access-wvklf\") pod \"maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6\" (UID: \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"
Oct 30 19:04:22.115274 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:22.115244    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6\" (UID: \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"
Oct 30 19:04:22.115633 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:22.115604    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6\" (UID: \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"
Oct 30 19:04:22.116539 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:22.116514    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6\" (UID: \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"
Oct 30 19:04:22.118439 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:22.118418    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6\" (UID: \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"
Oct 30 19:04:22.174058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:22.174020    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvklf\" (UniqueName: \"kubernetes.io/projected/2f7c308e-730e-4a27-b3c5-3a39071c66a3-kube-api-access-wvklf\") pod \"maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6\" (UID: \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"
Oct 30 19:04:22.174402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:22.174383    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"
Oct 30 19:04:22.621184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:22.619635    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"]
Oct 30 19:04:22.621184 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:04:22.620134    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7c308e_730e_4a27_b3c5_3a39071c66a3.slice/crio-1c8f7c79083668769f7d68af7601a4cb3bc7286a8ae99207ca9153caafca1f2a WatchSource:0}: Error finding container 1c8f7c79083668769f7d68af7601a4cb3bc7286a8ae99207ca9153caafca1f2a: Status 404 returned error can't find the container with id 1c8f7c79083668769f7d68af7601a4cb3bc7286a8ae99207ca9153caafca1f2a
Oct 30 19:04:23.058530 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:23.058497    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6" event={"ID":"2f7c308e-730e-4a27-b3c5-3a39071c66a3","Type":"ContainerStarted","Data":"1c8f7c79083668769f7d68af7601a4cb3bc7286a8ae99207ca9153caafca1f2a"}
Oct 30 19:04:23.260937 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:04:23.260899    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:04:23.528458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:23.528429    6014 scope.go:122] "RemoveContainer" containerID="724e31b3e39d2b53c80d58ac7ddaca8cedbb0dbc3133b40593e7cd16e147d563"
Oct 30 19:04:24.068734 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:24.068701    6014 generic.go:356] "Generic (PLEG): container finished" podID="2f7c308e-730e-4a27-b3c5-3a39071c66a3" containerID="7b912b22e2d19dc05a7d4857620e9616fbc05a64b20181a85482b70e4e7225f0" exitCode=0
Oct 30 19:04:24.069125 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:24.068741    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6" event={"ID":"2f7c308e-730e-4a27-b3c5-3a39071c66a3","Type":"ContainerDied","Data":"7b912b22e2d19dc05a7d4857620e9616fbc05a64b20181a85482b70e4e7225f0"}
Oct 30 19:04:25.281012 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:25.280985    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"
Oct 30 19:04:25.324105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:25.324058    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-xks-ca\") pod \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\" (UID: \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\") "
Oct 30 19:04:25.324370 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:25.324351    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-openshift-ca\") pod \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\" (UID: \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\") "
Oct 30 19:04:25.324827 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:25.324802    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-openshift-ca" pod "2f7c308e-730e-4a27-b3c5-3a39071c66a3" (UID: "2f7c308e-730e-4a27-b3c5-3a39071c66a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:25.324952 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:25.324934    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7c308e-730e-4a27-b3c5-3a39071c66a3-kube-api-access-wvklf\" (UniqueName: \"kubernetes.io/projected/2f7c308e-730e-4a27-b3c5-3a39071c66a3-kube-api-access-wvklf\") pod \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\" (UID: \"2f7c308e-730e-4a27-b3c5-3a39071c66a3\") "
Oct 30 19:04:25.325833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:25.325812    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.326744 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:25.326704    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-xks-ca" pod "2f7c308e-730e-4a27-b3c5-3a39071c66a3" (UID: "2f7c308e-730e-4a27-b3c5-3a39071c66a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:25.329326 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:25.329237    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7c308e-730e-4a27-b3c5-3a39071c66a3-kube-api-access-wvklf" pod "2f7c308e-730e-4a27-b3c5-3a39071c66a3" (UID: "2f7c308e-730e-4a27-b3c5-3a39071c66a3"). InnerVolumeSpecName "kube-api-access-wvklf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:25.428002 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:25.427958    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvklf\" (UniqueName: \"kubernetes.io/projected/2f7c308e-730e-4a27-b3c5-3a39071c66a3-kube-api-access-wvklf\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.428176 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:25.428009    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f7c308e-730e-4a27-b3c5-3a39071c66a3-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:26.087665 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:26.087624    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6" event={"ID":"2f7c308e-730e-4a27-b3c5-3a39071c66a3","Type":"ContainerDied","Data":"1c8f7c79083668769f7d68af7601a4cb3bc7286a8ae99207ca9153caafca1f2a"}
Oct 30 19:04:26.087860 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:26.087686    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c8f7c79083668769f7d68af7601a4cb3bc7286a8ae99207ca9153caafca1f2a"
Oct 30 19:04:26.087860 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:26.087638    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"
Oct 30 19:04:27.628909 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:27.628877    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:04:30.686164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:30.685710    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:30.911441 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:30.911374    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:31.117693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:31.117651    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:04:31.117883 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:31.117739    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" podUID="c9723156-3755-4457-8694-8ef4a234d191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:35.729889 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.729848    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"]
Oct 30 19:04:35.732203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.732117    6014 cpu_manager.go:414] "containerMap: removing container" podUID="2f7c308e-730e-4a27-b3c5-3a39071c66a3" containerName="revoke-keys"
Oct 30 19:04:35.732203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.732167    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7c308e-730e-4a27-b3c5-3a39071c66a3" containerName="revoke-keys"
Oct 30 19:04:35.732496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.732482    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7c308e-730e-4a27-b3c5-3a39071c66a3" containerName="revoke-keys"
Oct 30 19:04:35.737075 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.737048    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"
Oct 30 19:04:35.781205 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.781168    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"]
Oct 30 19:04:35.825194 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.819446    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:04:35.825194 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.823281    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:04:35.866857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.866819    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbg8c\" (UniqueName: \"kubernetes.io/projected/e93c132c-f785-4c22-a387-eaa5c5dc12f7-kube-api-access-cbg8c\") pod \"maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr\" (UID: \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"
Oct 30 19:04:35.867119 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.867095    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr\" (UID: \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"
Oct 30 19:04:35.867250 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.867225    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr\" (UID: \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"
Oct 30 19:04:35.969037 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.969002    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr\" (UID: \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"
Oct 30 19:04:35.969262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.969071    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr\" (UID: \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"
Oct 30 19:04:35.969337 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.969268    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbg8c\" (UniqueName: \"kubernetes.io/projected/e93c132c-f785-4c22-a387-eaa5c5dc12f7-kube-api-access-cbg8c\") pod \"maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr\" (UID: \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"
Oct 30 19:04:35.969740 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.969716    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr\" (UID: \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"
Oct 30 19:04:35.971977 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:35.971953    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr\" (UID: \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"
Oct 30 19:04:36.026477 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:36.026411    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbg8c\" (UniqueName: \"kubernetes.io/projected/e93c132c-f785-4c22-a387-eaa5c5dc12f7-kube-api-access-cbg8c\") pod \"maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr\" (UID: \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"
Oct 30 19:04:36.054105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:36.054072    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"
Oct 30 19:04:36.454078 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:04:36.454044    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93c132c_f785_4c22_a387_eaa5c5dc12f7.slice/crio-0572a747cc4793b61498b61f1a16141bf0faf0df3d92fab2b2d84241276ee245 WatchSource:0}: Error finding container 0572a747cc4793b61498b61f1a16141bf0faf0df3d92fab2b2d84241276ee245: Status 404 returned error can't find the container with id 0572a747cc4793b61498b61f1a16141bf0faf0df3d92fab2b2d84241276ee245
Oct 30 19:04:36.454431 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:36.454409    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"]
Oct 30 19:04:37.359357 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:37.359317    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr" event={"ID":"e93c132c-f785-4c22-a387-eaa5c5dc12f7","Type":"ContainerStarted","Data":"65c903b1dcb32bde1d63e916294fa2a5ec66f2d9e4f519f0b265ec57569970e0"}
Oct 30 19:04:37.359784 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:37.359390    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr" event={"ID":"e93c132c-f785-4c22-a387-eaa5c5dc12f7","Type":"ContainerStarted","Data":"0572a747cc4793b61498b61f1a16141bf0faf0df3d92fab2b2d84241276ee245"}
Oct 30 19:04:37.474446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:37.474396    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr" podStartSLOduration=2.474379983 podStartE2EDuration="2.474379983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:37.47105358 +0000 UTC m=+1516.114221196" watchObservedRunningTime="2026-10-30 19:04:37.474379983 +0000 UTC m=+1516.117547599"
Oct 30 19:04:38.369556 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:38.369517    6014 generic.go:356] "Generic (PLEG): container finished" podID="e93c132c-f785-4c22-a387-eaa5c5dc12f7" containerID="65c903b1dcb32bde1d63e916294fa2a5ec66f2d9e4f519f0b265ec57569970e0" exitCode=0
Oct 30 19:04:38.369949 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:38.369643    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr" event={"ID":"e93c132c-f785-4c22-a387-eaa5c5dc12f7","Type":"ContainerDied","Data":"65c903b1dcb32bde1d63e916294fa2a5ec66f2d9e4f519f0b265ec57569970e0"}
Oct 30 19:04:39.566727 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:39.566701    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"
Oct 30 19:04:39.734299 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:39.734216    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-xks-ca\") pod \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\" (UID: \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\") "
Oct 30 19:04:39.734479 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:39.734376    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-openshift-ca\") pod \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\" (UID: \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\") "
Oct 30 19:04:39.734479 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:39.734473    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93c132c-f785-4c22-a387-eaa5c5dc12f7-kube-api-access-cbg8c\" (UniqueName: \"kubernetes.io/projected/e93c132c-f785-4c22-a387-eaa5c5dc12f7-kube-api-access-cbg8c\") pod \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\" (UID: \"e93c132c-f785-4c22-a387-eaa5c5dc12f7\") "
Oct 30 19:04:39.734800 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:39.734775    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-openshift-ca" pod "e93c132c-f785-4c22-a387-eaa5c5dc12f7" (UID: "e93c132c-f785-4c22-a387-eaa5c5dc12f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:39.735335 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:39.735312    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:39.737107 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:39.737073    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-xks-ca" pod "e93c132c-f785-4c22-a387-eaa5c5dc12f7" (UID: "e93c132c-f785-4c22-a387-eaa5c5dc12f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:39.737349 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:39.737323    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93c132c-f785-4c22-a387-eaa5c5dc12f7-kube-api-access-cbg8c" pod "e93c132c-f785-4c22-a387-eaa5c5dc12f7" (UID: "e93c132c-f785-4c22-a387-eaa5c5dc12f7"). InnerVolumeSpecName "kube-api-access-cbg8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:39.837372 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:39.837317    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbg8c\" (UniqueName: \"kubernetes.io/projected/e93c132c-f785-4c22-a387-eaa5c5dc12f7-kube-api-access-cbg8c\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:39.837372 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:39.837364    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93c132c-f785-4c22-a387-eaa5c5dc12f7-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:40.388758 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:40.388724    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr" event={"ID":"e93c132c-f785-4c22-a387-eaa5c5dc12f7","Type":"ContainerDied","Data":"0572a747cc4793b61498b61f1a16141bf0faf0df3d92fab2b2d84241276ee245"}
Oct 30 19:04:40.388758 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:40.388753    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"
Oct 30 19:04:40.388968 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:40.388775    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0572a747cc4793b61498b61f1a16141bf0faf0df3d92fab2b2d84241276ee245"
Oct 30 19:04:43.583538 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.583504    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"]
Oct 30 19:04:43.584371 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.584353    6014 cpu_manager.go:414] "containerMap: removing container" podUID="e93c132c-f785-4c22-a387-eaa5c5dc12f7" containerName="revoke-keys"
Oct 30 19:04:43.584428 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.584373    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93c132c-f785-4c22-a387-eaa5c5dc12f7" containerName="revoke-keys"
Oct 30 19:04:43.584577 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.584566    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93c132c-f785-4c22-a387-eaa5c5dc12f7" containerName="revoke-keys"
Oct 30 19:04:43.587511 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.587492    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"
Oct 30 19:04:43.615319 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.615289    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2be7d8a7-4d5rr\" (UID: \"dd62bd5a-0559-41b4-b504-901d6b32782f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"
Oct 30 19:04:43.615462 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.615334    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2be7d8a7-4d5rr\" (UID: \"dd62bd5a-0559-41b4-b504-901d6b32782f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"
Oct 30 19:04:43.615511 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.615468    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92q22\" (UniqueName: \"kubernetes.io/projected/dd62bd5a-0559-41b4-b504-901d6b32782f-kube-api-access-92q22\") pod \"maas-api-revoke-sub-2be7d8a7-4d5rr\" (UID: \"dd62bd5a-0559-41b4-b504-901d6b32782f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"
Oct 30 19:04:43.646896 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.646865    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"]
Oct 30 19:04:43.688945 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.688914    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:04:43.689197 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.688953    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:04:43.717070 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.717039    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2be7d8a7-4d5rr\" (UID: \"dd62bd5a-0559-41b4-b504-901d6b32782f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"
Oct 30 19:04:43.717267 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.717084    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2be7d8a7-4d5rr\" (UID: \"dd62bd5a-0559-41b4-b504-901d6b32782f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"
Oct 30 19:04:43.717267 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.717183    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92q22\" (UniqueName: \"kubernetes.io/projected/dd62bd5a-0559-41b4-b504-901d6b32782f-kube-api-access-92q22\") pod \"maas-api-revoke-sub-2be7d8a7-4d5rr\" (UID: \"dd62bd5a-0559-41b4-b504-901d6b32782f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"
Oct 30 19:04:43.717707 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.717688    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2be7d8a7-4d5rr\" (UID: \"dd62bd5a-0559-41b4-b504-901d6b32782f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"
Oct 30 19:04:43.719655 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.719638    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2be7d8a7-4d5rr\" (UID: \"dd62bd5a-0559-41b4-b504-901d6b32782f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"
Oct 30 19:04:43.774083 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.774050    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92q22\" (UniqueName: \"kubernetes.io/projected/dd62bd5a-0559-41b4-b504-901d6b32782f-kube-api-access-92q22\") pod \"maas-api-revoke-sub-2be7d8a7-4d5rr\" (UID: \"dd62bd5a-0559-41b4-b504-901d6b32782f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"
Oct 30 19:04:43.901012 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:43.900927    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"
Oct 30 19:04:44.284578 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:44.284547    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"]
Oct 30 19:04:44.284722 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:04:44.284694    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd62bd5a_0559_41b4_b504_901d6b32782f.slice/crio-4b6ab8e762840abe39d8ba823ca7bac2d6563929dc673b36d6ba19c20ecce69a WatchSource:0}: Error finding container 4b6ab8e762840abe39d8ba823ca7bac2d6563929dc673b36d6ba19c20ecce69a: Status 404 returned error can't find the container with id 4b6ab8e762840abe39d8ba823ca7bac2d6563929dc673b36d6ba19c20ecce69a
Oct 30 19:04:44.422761 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:44.422723    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr" event={"ID":"dd62bd5a-0559-41b4-b504-901d6b32782f","Type":"ContainerStarted","Data":"4b6ab8e762840abe39d8ba823ca7bac2d6563929dc673b36d6ba19c20ecce69a"}
Oct 30 19:04:45.432703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:45.432671    6014 generic.go:356] "Generic (PLEG): container finished" podID="dd62bd5a-0559-41b4-b504-901d6b32782f" containerID="f4fbc7657f0fe20debea765c6321f6d120cd471efab544ec06ea51df7204631e" exitCode=0
Oct 30 19:04:45.433120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:45.432756    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr" event={"ID":"dd62bd5a-0559-41b4-b504-901d6b32782f","Type":"ContainerDied","Data":"f4fbc7657f0fe20debea765c6321f6d120cd471efab544ec06ea51df7204631e"}
Oct 30 19:04:46.523658 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:46.523235    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"]
Oct 30 19:04:46.615668 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:46.615642    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"
Oct 30 19:04:46.632317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:46.632291    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"]
Oct 30 19:04:46.663317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:46.663278    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-xks-ca\") pod \"dd62bd5a-0559-41b4-b504-901d6b32782f\" (UID: \"dd62bd5a-0559-41b4-b504-901d6b32782f\") "
Oct 30 19:04:46.663525 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:46.663508    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd62bd5a-0559-41b4-b504-901d6b32782f-kube-api-access-92q22\" (UniqueName: \"kubernetes.io/projected/dd62bd5a-0559-41b4-b504-901d6b32782f-kube-api-access-92q22\") pod \"dd62bd5a-0559-41b4-b504-901d6b32782f\" (UID: \"dd62bd5a-0559-41b4-b504-901d6b32782f\") "
Oct 30 19:04:46.663591 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:46.663565    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-openshift-ca\") pod \"dd62bd5a-0559-41b4-b504-901d6b32782f\" (UID: \"dd62bd5a-0559-41b4-b504-901d6b32782f\") "
Oct 30 19:04:46.663877 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:46.663855    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-openshift-ca" pod "dd62bd5a-0559-41b4-b504-901d6b32782f" (UID: "dd62bd5a-0559-41b4-b504-901d6b32782f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:46.664284 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:46.664263    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:46.665547 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:46.665512    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-xks-ca" pod "dd62bd5a-0559-41b4-b504-901d6b32782f" (UID: "dd62bd5a-0559-41b4-b504-901d6b32782f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:46.665688 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:46.665673    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd62bd5a-0559-41b4-b504-901d6b32782f-kube-api-access-92q22" pod "dd62bd5a-0559-41b4-b504-901d6b32782f" (UID: "dd62bd5a-0559-41b4-b504-901d6b32782f"). InnerVolumeSpecName "kube-api-access-92q22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:46.766107 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:46.766011    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd62bd5a-0559-41b4-b504-901d6b32782f-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:46.766485 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:46.766458    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92q22\" (UniqueName: \"kubernetes.io/projected/dd62bd5a-0559-41b4-b504-901d6b32782f-kube-api-access-92q22\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:47.453426 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:47.453391    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr" event={"ID":"dd62bd5a-0559-41b4-b504-901d6b32782f","Type":"ContainerDied","Data":"4b6ab8e762840abe39d8ba823ca7bac2d6563929dc673b36d6ba19c20ecce69a"}
Oct 30 19:04:47.453426 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:47.453414    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"
Oct 30 19:04:47.453650 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:47.453447    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b6ab8e762840abe39d8ba823ca7bac2d6563929dc673b36d6ba19c20ecce69a"
Oct 30 19:04:51.524361 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.524299    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4" podUID="3eeea6a3-87d6-4299-98f9-23cc4f27a912" containerName="payload-processing" containerID="cri-o://b88805ddf5b09a5dee8aa88ba210d78e33537df78d783bb5eb387bd951860a58" gracePeriod=25
Oct 30 19:04:51.633441 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.633388    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm" podUID="b976ece9-9e38-4094-8d29-5845669b59a9" containerName="payload-pre-processing" containerID="cri-o://b17b6265c84170422b58de8a9ddeec7c4dfbf2dd341d821012ba5963fa30d487" gracePeriod=25
Oct 30 19:04:51.848272 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.848239    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"
Oct 30 19:04:51.894060 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.894007    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eeea6a3-87d6-4299-98f9-23cc4f27a912-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eeea6a3-87d6-4299-98f9-23cc4f27a912-plugins-config-volume\") pod \"3eeea6a3-87d6-4299-98f9-23cc4f27a912\" (UID: \"3eeea6a3-87d6-4299-98f9-23cc4f27a912\") "
Oct 30 19:04:51.894060 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.894055    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eeea6a3-87d6-4299-98f9-23cc4f27a912-kube-api-access-4pfsl\" (UniqueName: \"kubernetes.io/projected/3eeea6a3-87d6-4299-98f9-23cc4f27a912-kube-api-access-4pfsl\") pod \"3eeea6a3-87d6-4299-98f9-23cc4f27a912\" (UID: \"3eeea6a3-87d6-4299-98f9-23cc4f27a912\") "
Oct 30 19:04:51.894476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.894411    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eeea6a3-87d6-4299-98f9-23cc4f27a912-plugins-config-volume" pod "3eeea6a3-87d6-4299-98f9-23cc4f27a912" (UID: "3eeea6a3-87d6-4299-98f9-23cc4f27a912"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:51.895088 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.895052    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eeea6a3-87d6-4299-98f9-23cc4f27a912-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:51.896607 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.896580    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eeea6a3-87d6-4299-98f9-23cc4f27a912-kube-api-access-4pfsl" pod "3eeea6a3-87d6-4299-98f9-23cc4f27a912" (UID: "3eeea6a3-87d6-4299-98f9-23cc4f27a912"). InnerVolumeSpecName "kube-api-access-4pfsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:51.923852 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.923827    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"
Oct 30 19:04:51.995957 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.995927    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b976ece9-9e38-4094-8d29-5845669b59a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b976ece9-9e38-4094-8d29-5845669b59a9-plugins-config-volume\") pod \"b976ece9-9e38-4094-8d29-5845669b59a9\" (UID: \"b976ece9-9e38-4094-8d29-5845669b59a9\") "
Oct 30 19:04:51.996185 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.996034    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b976ece9-9e38-4094-8d29-5845669b59a9-kube-api-access-f7znj\" (UniqueName: \"kubernetes.io/projected/b976ece9-9e38-4094-8d29-5845669b59a9-kube-api-access-f7znj\") pod \"b976ece9-9e38-4094-8d29-5845669b59a9\" (UID: \"b976ece9-9e38-4094-8d29-5845669b59a9\") "
Oct 30 19:04:51.996382 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.996356    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b976ece9-9e38-4094-8d29-5845669b59a9-plugins-config-volume" pod "b976ece9-9e38-4094-8d29-5845669b59a9" (UID: "b976ece9-9e38-4094-8d29-5845669b59a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:51.996859 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.996841    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pfsl\" (UniqueName: \"kubernetes.io/projected/3eeea6a3-87d6-4299-98f9-23cc4f27a912-kube-api-access-4pfsl\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:51.996939 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.996864    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b976ece9-9e38-4094-8d29-5845669b59a9-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:51.998421 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:51.998397    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b976ece9-9e38-4094-8d29-5845669b59a9-kube-api-access-f7znj" pod "b976ece9-9e38-4094-8d29-5845669b59a9" (UID: "b976ece9-9e38-4094-8d29-5845669b59a9"). InnerVolumeSpecName "kube-api-access-f7znj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:52.099040 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.098994    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7znj\" (UniqueName: \"kubernetes.io/projected/b976ece9-9e38-4094-8d29-5845669b59a9-kube-api-access-f7znj\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.501221 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.501102    6014 generic.go:356] "Generic (PLEG): container finished" podID="3eeea6a3-87d6-4299-98f9-23cc4f27a912" containerID="b88805ddf5b09a5dee8aa88ba210d78e33537df78d783bb5eb387bd951860a58" exitCode=0
Oct 30 19:04:52.501221 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.501177    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4" event={"ID":"3eeea6a3-87d6-4299-98f9-23cc4f27a912","Type":"ContainerDied","Data":"b88805ddf5b09a5dee8aa88ba210d78e33537df78d783bb5eb387bd951860a58"}
Oct 30 19:04:52.501221 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.501201    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"
Oct 30 19:04:52.501489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.501233    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4" event={"ID":"3eeea6a3-87d6-4299-98f9-23cc4f27a912","Type":"ContainerDied","Data":"5d0d6b07a46728dbb5a4dcd1f8f0e2b262efe27ac4c342af13a4ca07aad9501f"}
Oct 30 19:04:52.501489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.501262    6014 scope.go:122] "RemoveContainer" containerID="b88805ddf5b09a5dee8aa88ba210d78e33537df78d783bb5eb387bd951860a58"
Oct 30 19:04:52.503492 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.503463    6014 generic.go:356] "Generic (PLEG): container finished" podID="b976ece9-9e38-4094-8d29-5845669b59a9" containerID="b17b6265c84170422b58de8a9ddeec7c4dfbf2dd341d821012ba5963fa30d487" exitCode=0
Oct 30 19:04:52.503635 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.503506    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"
Oct 30 19:04:52.503635 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.503523    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm" event={"ID":"b976ece9-9e38-4094-8d29-5845669b59a9","Type":"ContainerDied","Data":"b17b6265c84170422b58de8a9ddeec7c4dfbf2dd341d821012ba5963fa30d487"}
Oct 30 19:04:52.503635 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.503561    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm" event={"ID":"b976ece9-9e38-4094-8d29-5845669b59a9","Type":"ContainerDied","Data":"0ba64687e56922060e80d60cd06c9c775662544a1c06250e685f4a6f8409af9a"}
Oct 30 19:04:52.514550 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.514531    6014 scope.go:122] "RemoveContainer" containerID="b88805ddf5b09a5dee8aa88ba210d78e33537df78d783bb5eb387bd951860a58"
Oct 30 19:04:52.514874 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:04:52.514855    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b88805ddf5b09a5dee8aa88ba210d78e33537df78d783bb5eb387bd951860a58\": container with ID starting with b88805ddf5b09a5dee8aa88ba210d78e33537df78d783bb5eb387bd951860a58 not found: ID does not exist" containerID="b88805ddf5b09a5dee8aa88ba210d78e33537df78d783bb5eb387bd951860a58"
Oct 30 19:04:52.514950 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.514882    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88805ddf5b09a5dee8aa88ba210d78e33537df78d783bb5eb387bd951860a58"} err="failed to get container status \"b88805ddf5b09a5dee8aa88ba210d78e33537df78d783bb5eb387bd951860a58\": rpc error: code = NotFound desc = could not find container \"b88805ddf5b09a5dee8aa88ba210d78e33537df78d783bb5eb387bd951860a58\": container with ID starting with b88805ddf5b09a5dee8aa88ba210d78e33537df78d783bb5eb387bd951860a58 not found: ID does not exist"
Oct 30 19:04:52.514950 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.514899    6014 scope.go:122] "RemoveContainer" containerID="b17b6265c84170422b58de8a9ddeec7c4dfbf2dd341d821012ba5963fa30d487"
Oct 30 19:04:52.525472 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.525203    6014 scope.go:122] "RemoveContainer" containerID="b17b6265c84170422b58de8a9ddeec7c4dfbf2dd341d821012ba5963fa30d487"
Oct 30 19:04:52.525711 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:04:52.525500    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17b6265c84170422b58de8a9ddeec7c4dfbf2dd341d821012ba5963fa30d487\": container with ID starting with b17b6265c84170422b58de8a9ddeec7c4dfbf2dd341d821012ba5963fa30d487 not found: ID does not exist" containerID="b17b6265c84170422b58de8a9ddeec7c4dfbf2dd341d821012ba5963fa30d487"
Oct 30 19:04:52.525711 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.525529    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17b6265c84170422b58de8a9ddeec7c4dfbf2dd341d821012ba5963fa30d487"} err="failed to get container status \"b17b6265c84170422b58de8a9ddeec7c4dfbf2dd341d821012ba5963fa30d487\": rpc error: code = NotFound desc = could not find container \"b17b6265c84170422b58de8a9ddeec7c4dfbf2dd341d821012ba5963fa30d487\": container with ID starting with b17b6265c84170422b58de8a9ddeec7c4dfbf2dd341d821012ba5963fa30d487 not found: ID does not exist"
Oct 30 19:04:52.668336 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.668296    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"]
Oct 30 19:04:52.672838 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.672811    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cd57e7dd-56486cb959-8sllm"]
Oct 30 19:04:52.777492 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.777375    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"]
Oct 30 19:04:52.781688 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:52.781660    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cd57e7dd-755b46b697-77hr4"]
Oct 30 19:04:53.260512 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:04:53.260471    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:04:53.478241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:53.478214    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eeea6a3-87d6-4299-98f9-23cc4f27a912" path="/var/lib/kubelet/pods/3eeea6a3-87d6-4299-98f9-23cc4f27a912/volumes"
Oct 30 19:04:53.478676 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:04:53.478662    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b976ece9-9e38-4094-8d29-5845669b59a9" path="/var/lib/kubelet/pods/b976ece9-9e38-4094-8d29-5845669b59a9/volumes"
Oct 30 19:05:06.761684 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:06.761610    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:05:18.631928 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:18.631896    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"]
Oct 30 19:05:18.683201 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:18.683165    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cfaf8925-5vzl6"]
Oct 30 19:05:18.709562 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:18.709529    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"]
Oct 30 19:05:18.709814 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:18.709789    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s" podUID="681008f7-dc77-453a-a264-fd69fc0fe590" containerName="istio-proxy" containerID="cri-o://b5c714fd8f7ea617153832fd1428f8d3237dc0b83e8f094ea2500ab8379c431d" gracePeriod=30
Oct 30 19:05:19.477857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:19.477825    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7c308e-730e-4a27-b3c5-3a39071c66a3" path="/var/lib/kubelet/pods/2f7c308e-730e-4a27-b3c5-3a39071c66a3/volumes"
Oct 30 19:05:20.060236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:20.060198    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"]
Oct 30 19:05:20.108274 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:20.108238    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef5529e-bb5nj"]
Oct 30 19:05:21.478759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:21.478728    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c903929a-5bc5-426e-9bcc-caa2fe0e0d2d" path="/var/lib/kubelet/pods/c903929a-5bc5-426e-9bcc-caa2fe0e0d2d/volumes"
Oct 30 19:05:23.261811 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:05:23.261765    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:05:23.847010 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:23.846970    6014 generic.go:356] "Generic (PLEG): container finished" podID="681008f7-dc77-453a-a264-fd69fc0fe590" containerID="b5c714fd8f7ea617153832fd1428f8d3237dc0b83e8f094ea2500ab8379c431d" exitCode=0
Oct 30 19:05:23.847201 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:23.847033    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s" event={"ID":"681008f7-dc77-453a-a264-fd69fc0fe590","Type":"ContainerDied","Data":"b5c714fd8f7ea617153832fd1428f8d3237dc0b83e8f094ea2500ab8379c431d"}
Oct 30 19:05:23.847201 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:23.847099    6014 scope.go:122] "RemoveContainer" containerID="3fa63dc51d111f3f2668905339eab3bc7321e77a6a96fcbebd806ba5a633e713"
Oct 30 19:05:24.021816 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.021788    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:05:24.025835 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.025806    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-certs\") pod \"681008f7-dc77-453a-a264-fd69fc0fe590\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") "
Oct 30 19:05:24.025991 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.025877    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681008f7-dc77-453a-a264-fd69fc0fe590-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681008f7-dc77-453a-a264-fd69fc0fe590-istiod-ca-cert\") pod \"681008f7-dc77-453a-a264-fd69fc0fe590\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") "
Oct 30 19:05:24.025991 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.025958    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-data\") pod \"681008f7-dc77-453a-a264-fd69fc0fe590\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") "
Oct 30 19:05:24.026116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.026043    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-envoy\") pod \"681008f7-dc77-453a-a264-fd69fc0fe590\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") "
Oct 30 19:05:24.026116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.026049    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-certs" pod "681008f7-dc77-453a-a264-fd69fc0fe590" (UID: "681008f7-dc77-453a-a264-fd69fc0fe590"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:24.026116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.026085    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-istio-token\" (UniqueName: \"kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-istio-token\") pod \"681008f7-dc77-453a-a264-fd69fc0fe590\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") "
Oct 30 19:05:24.026293 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.026164    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-kube-api-access-nwkb2\" (UniqueName: \"kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-kube-api-access-nwkb2\") pod \"681008f7-dc77-453a-a264-fd69fc0fe590\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") "
Oct 30 19:05:24.026293 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.026216    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/681008f7-dc77-453a-a264-fd69fc0fe590-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681008f7-dc77-453a-a264-fd69fc0fe590-istio-podinfo\") pod \"681008f7-dc77-453a-a264-fd69fc0fe590\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") "
Oct 30 19:05:24.026293 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.026233    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-data" pod "681008f7-dc77-453a-a264-fd69fc0fe590" (UID: "681008f7-dc77-453a-a264-fd69fc0fe590"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:24.026293 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.026269    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-credential-socket\") pod \"681008f7-dc77-453a-a264-fd69fc0fe590\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") "
Oct 30 19:05:24.026293 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.026271    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681008f7-dc77-453a-a264-fd69fc0fe590-istiod-ca-cert" pod "681008f7-dc77-453a-a264-fd69fc0fe590" (UID: "681008f7-dc77-453a-a264-fd69fc0fe590"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:24.026540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.026328    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-socket\") pod \"681008f7-dc77-453a-a264-fd69fc0fe590\" (UID: \"681008f7-dc77-453a-a264-fd69fc0fe590\") "
Oct 30 19:05:24.026695 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.026675    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-credential-socket" pod "681008f7-dc77-453a-a264-fd69fc0fe590" (UID: "681008f7-dc77-453a-a264-fd69fc0fe590"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:24.026942 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.026902    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-socket" pod "681008f7-dc77-453a-a264-fd69fc0fe590" (UID: "681008f7-dc77-453a-a264-fd69fc0fe590"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:24.027050 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.027031    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:24.027114 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.027057    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681008f7-dc77-453a-a264-fd69fc0fe590-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:24.027114 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.027071    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:24.027114 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.027086    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:24.027114 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.027099    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:24.028571 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.028542    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-envoy" pod "681008f7-dc77-453a-a264-fd69fc0fe590" (UID: "681008f7-dc77-453a-a264-fd69fc0fe590"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:24.028571 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.028553    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-kube-api-access-nwkb2" pod "681008f7-dc77-453a-a264-fd69fc0fe590" (UID: "681008f7-dc77-453a-a264-fd69fc0fe590"). InnerVolumeSpecName "kube-api-access-nwkb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:24.028905 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.028891    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-istio-token" pod "681008f7-dc77-453a-a264-fd69fc0fe590" (UID: "681008f7-dc77-453a-a264-fd69fc0fe590"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:24.029063 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.029035    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/681008f7-dc77-453a-a264-fd69fc0fe590-istio-podinfo" pod "681008f7-dc77-453a-a264-fd69fc0fe590" (UID: "681008f7-dc77-453a-a264-fd69fc0fe590"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:05:24.129077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.128988    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwkb2\" (UniqueName: \"kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-kube-api-access-nwkb2\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:24.129077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.129018    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681008f7-dc77-453a-a264-fd69fc0fe590-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:24.129077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.129029    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681008f7-dc77-453a-a264-fd69fc0fe590-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:24.129077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.129037    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681008f7-dc77-453a-a264-fd69fc0fe590-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:24.856285 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.856257    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"
Oct 30 19:05:24.856285 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.856275    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s" event={"ID":"681008f7-dc77-453a-a264-fd69fc0fe590","Type":"ContainerDied","Data":"2949765fe5e401a44d35deee7c879f6a50a8dddc2a9e0b80adfc7ed41f177a71"}
Oct 30 19:05:24.856812 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:24.856338    6014 scope.go:122] "RemoveContainer" containerID="b5c714fd8f7ea617153832fd1428f8d3237dc0b83e8f094ea2500ab8379c431d"
Oct 30 19:05:25.072341 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:25.072308    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"]
Oct 30 19:05:25.121585 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:25.121503    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cfaf8925-gw-openshift-default-85468459bf-7rd2s"]
Oct 30 19:05:25.477762 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:25.477683    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681008f7-dc77-453a-a264-fd69fc0fe590" path="/var/lib/kubelet/pods/681008f7-dc77-453a-a264-fd69fc0fe590/volumes"
Oct 30 19:05:27.084962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.084841    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"]
Oct 30 19:05:27.115107 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.115054    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f86ed71-nvmrd"]
Oct 30 19:05:27.510319 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.510165    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="291abfe2-8f65-42e1-a425-957ac2a2c1d4" path="/var/lib/kubelet/pods/291abfe2-8f65-42e1-a425-957ac2a2c1d4/volumes"
Oct 30 19:05:27.723862 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.723824    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"]
Oct 30 19:05:27.725028 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725001    6014 cpu_manager.go:414] "containerMap: removing container" podUID="dd62bd5a-0559-41b4-b504-901d6b32782f" containerName="revoke-keys"
Oct 30 19:05:27.725028 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725029    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd62bd5a-0559-41b4-b504-901d6b32782f" containerName="revoke-keys"
Oct 30 19:05:27.725353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725089    6014 cpu_manager.go:414] "containerMap: removing container" podUID="681008f7-dc77-453a-a264-fd69fc0fe590" containerName="istio-proxy"
Oct 30 19:05:27.725353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725105    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681008f7-dc77-453a-a264-fd69fc0fe590" containerName="istio-proxy"
Oct 30 19:05:27.725353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725129    6014 cpu_manager.go:414] "containerMap: removing container" podUID="3eeea6a3-87d6-4299-98f9-23cc4f27a912" containerName="payload-processing"
Oct 30 19:05:27.725353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725156    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eeea6a3-87d6-4299-98f9-23cc4f27a912" containerName="payload-processing"
Oct 30 19:05:27.725353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725170    6014 cpu_manager.go:414] "containerMap: removing container" podUID="681008f7-dc77-453a-a264-fd69fc0fe590" containerName="istio-proxy"
Oct 30 19:05:27.725353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725177    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681008f7-dc77-453a-a264-fd69fc0fe590" containerName="istio-proxy"
Oct 30 19:05:27.725353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725229    6014 cpu_manager.go:414] "containerMap: removing container" podUID="b976ece9-9e38-4094-8d29-5845669b59a9" containerName="payload-pre-processing"
Oct 30 19:05:27.725353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725240    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b976ece9-9e38-4094-8d29-5845669b59a9" containerName="payload-pre-processing"
Oct 30 19:05:27.725599 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725447    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b976ece9-9e38-4094-8d29-5845669b59a9" containerName="payload-pre-processing"
Oct 30 19:05:27.725599 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725467    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd62bd5a-0559-41b4-b504-901d6b32782f" containerName="revoke-keys"
Oct 30 19:05:27.725599 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725487    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="681008f7-dc77-453a-a264-fd69fc0fe590" containerName="istio-proxy"
Oct 30 19:05:27.725599 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725498    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eeea6a3-87d6-4299-98f9-23cc4f27a912" containerName="payload-processing"
Oct 30 19:05:27.725599 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.725506    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="681008f7-dc77-453a-a264-fd69fc0fe590" containerName="istio-proxy"
Oct 30 19:05:27.730588 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.730254    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"
Oct 30 19:05:27.784133 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.784048    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-eb4c249f\""
Oct 30 19:05:27.796231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.793889    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"]
Oct 30 19:05:27.836198 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.828127    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-eb4c249f-dockercfg-kv74j\""
Oct 30 19:05:27.905548 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.905515    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99qhw\" (UniqueName: \"kubernetes.io/projected/bf265f8d-4e38-44b7-aba7-e7e7716f6579-kube-api-access-99qhw\") pod \"payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5\" (UID: \"bf265f8d-4e38-44b7-aba7-e7e7716f6579\") " pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"
Oct 30 19:05:27.905703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:27.905577    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf265f8d-4e38-44b7-aba7-e7e7716f6579-plugins-config-volume\") pod \"payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5\" (UID: \"bf265f8d-4e38-44b7-aba7-e7e7716f6579\") " pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"
Oct 30 19:05:28.013376 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:28.013341    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99qhw\" (UniqueName: \"kubernetes.io/projected/bf265f8d-4e38-44b7-aba7-e7e7716f6579-kube-api-access-99qhw\") pod \"payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5\" (UID: \"bf265f8d-4e38-44b7-aba7-e7e7716f6579\") " pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"
Oct 30 19:05:28.013582 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:28.013426    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf265f8d-4e38-44b7-aba7-e7e7716f6579-plugins-config-volume\") pod \"payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5\" (UID: \"bf265f8d-4e38-44b7-aba7-e7e7716f6579\") " pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"
Oct 30 19:05:28.014783 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:28.014752    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf265f8d-4e38-44b7-aba7-e7e7716f6579-plugins-config-volume\") pod \"payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5\" (UID: \"bf265f8d-4e38-44b7-aba7-e7e7716f6579\") " pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"
Oct 30 19:05:28.073906 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:28.073874    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99qhw\" (UniqueName: \"kubernetes.io/projected/bf265f8d-4e38-44b7-aba7-e7e7716f6579-kube-api-access-99qhw\") pod \"payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5\" (UID: \"bf265f8d-4e38-44b7-aba7-e7e7716f6579\") " pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"
Oct 30 19:05:28.354529 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:28.354439    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"
Oct 30 19:05:28.738109 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:05:28.738070    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf265f8d_4e38_44b7_aba7_e7e7716f6579.slice/crio-7e744e980fedb24c15a48dbb4cde317f5e8c7e4fe4a28c38cd472183e4433385 WatchSource:0}: Error finding container 7e744e980fedb24c15a48dbb4cde317f5e8c7e4fe4a28c38cd472183e4433385: Status 404 returned error can't find the container with id 7e744e980fedb24c15a48dbb4cde317f5e8c7e4fe4a28c38cd472183e4433385
Oct 30 19:05:28.738328 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:28.738300    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"]
Oct 30 19:05:28.908759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:28.908715    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5" event={"ID":"bf265f8d-4e38-44b7-aba7-e7e7716f6579","Type":"ContainerStarted","Data":"7e744e980fedb24c15a48dbb4cde317f5e8c7e4fe4a28c38cd472183e4433385"}
Oct 30 19:05:29.919342 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:29.919305    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5" event={"ID":"bf265f8d-4e38-44b7-aba7-e7e7716f6579","Type":"ContainerStarted","Data":"2d5623f10a48f84655e2e20707365fcb8d58d1d7e155f7afb15a73e25f5c5243"}
Oct 30 19:05:29.919783 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:29.919394    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"
Oct 30 19:05:30.036392 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:30.036336    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5" podStartSLOduration=3.036317019 podStartE2EDuration="3.036317019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:30.032469095 +0000 UTC m=+1568.675636712" watchObservedRunningTime="2026-10-30 19:05:30.036317019 +0000 UTC m=+1568.679484635"
Oct 30 19:05:30.706798 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:30.706768    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"]
Oct 30 19:05:30.710720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:30.710698    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cd57e7dd-7vcsr"]
Oct 30 19:05:30.763763 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:30.763727    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"]
Oct 30 19:05:30.764046 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:30.764022    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw" podUID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerName="istio-proxy" containerID="cri-o://bd810839730995db5de0fd221120a5bd4719528a44e150ecd3d7520d89b70480" gracePeriod=30
Oct 30 19:05:31.044224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.043901    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"]
Oct 30 19:05:31.064080 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.064042    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.101130 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.100403    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"]
Oct 30 19:05:31.190502 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.190464    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkf8d\" (UniqueName: \"kubernetes.io/projected/aacb99a2-5355-4e64-86bb-ad338415035f-kube-api-access-jkf8d\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.190684 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.190525    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.190684 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.190636    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.190787 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.190769    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacb99a2-5355-4e64-86bb-ad338415035f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.190842 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.190830    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacb99a2-5355-4e64-86bb-ad338415035f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.190937 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.190921    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.190999 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.190951    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacb99a2-5355-4e64-86bb-ad338415035f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.191076 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.191063    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.191273 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.191256    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.292847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.292810    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacb99a2-5355-4e64-86bb-ad338415035f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.292847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.292857    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.293120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.292872    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacb99a2-5355-4e64-86bb-ad338415035f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.293120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.292915    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.293120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.293012    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.293120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.293077    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkf8d\" (UniqueName: \"kubernetes.io/projected/aacb99a2-5355-4e64-86bb-ad338415035f-kube-api-access-jkf8d\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.293120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.293117    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.293479 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.293187    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.293479 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.293287    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.293479 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.293346    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacb99a2-5355-4e64-86bb-ad338415035f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.293479 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.293372    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.293857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.293566    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.293857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.293566    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.293968 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.293881    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacb99a2-5355-4e64-86bb-ad338415035f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.295820 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.295760    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacb99a2-5355-4e64-86bb-ad338415035f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.295940 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.295845    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacb99a2-5355-4e64-86bb-ad338415035f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.349607 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.349576    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacb99a2-5355-4e64-86bb-ad338415035f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.349833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.349811    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkf8d\" (UniqueName: \"kubernetes.io/projected/aacb99a2-5355-4e64-86bb-ad338415035f-kube-api-access-jkf8d\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dmh57\" (UID: \"aacb99a2-5355-4e64-86bb-ad338415035f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.388391 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.388351    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:31.481211 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.481179    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93c132c-f785-4c22-a387-eaa5c5dc12f7" path="/var/lib/kubelet/pods/e93c132c-f785-4c22-a387-eaa5c5dc12f7/volumes"
Oct 30 19:05:31.827563 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:05:31.827526    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacb99a2_5355_4e64_86bb_ad338415035f.slice/crio-efcf42d54a6b24bc8e930b06b29e92fbe1ba6c66e1df9e9b1b94547f5eb0e00c WatchSource:0}: Error finding container efcf42d54a6b24bc8e930b06b29e92fbe1ba6c66e1df9e9b1b94547f5eb0e00c: Status 404 returned error can't find the container with id efcf42d54a6b24bc8e930b06b29e92fbe1ba6c66e1df9e9b1b94547f5eb0e00c
Oct 30 19:05:31.827868 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.827842    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"]
Oct 30 19:05:31.830692 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.830656    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:05:31.830774 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.830725    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:05:31.830774 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:31.830754    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:05:32.097239 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:32.080377    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"]
Oct 30 19:05:32.097239 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:32.089489    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d084ade5-bpx6s"]
Oct 30 19:05:32.100311 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:32.100269    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" event={"ID":"aacb99a2-5355-4e64-86bb-ad338415035f","Type":"ContainerStarted","Data":"e0974260515149fa0de864b257f06bdb6a94edf8df72261757bb55c02bf79fe7"}
Oct 30 19:05:32.100477 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:32.100335    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" event={"ID":"aacb99a2-5355-4e64-86bb-ad338415035f","Type":"ContainerStarted","Data":"efcf42d54a6b24bc8e930b06b29e92fbe1ba6c66e1df9e9b1b94547f5eb0e00c"}
Oct 30 19:05:32.227482 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:32.227418    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" podStartSLOduration=2.227402185 podStartE2EDuration="2.227402185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:32.223154316 +0000 UTC m=+1570.866321928" watchObservedRunningTime="2026-10-30 19:05:32.227402185 +0000 UTC m=+1570.870569799"
Oct 30 19:05:32.389374 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:32.389277    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:32.391401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:32.391371    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:05:32.391543 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:32.391441    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" podUID="aacb99a2-5355-4e64-86bb-ad338415035f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 19:05:33.400166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:33.394285    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:05:33.400166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:33.394359    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" podUID="aacb99a2-5355-4e64-86bb-ad338415035f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 19:05:33.516185 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:33.512731    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b1d03a-2932-4f68-8ef1-b35960cd7daf" path="/var/lib/kubelet/pods/51b1d03a-2932-4f68-8ef1-b35960cd7daf/volumes"
Oct 30 19:05:34.033777 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.033745    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"]
Oct 30 19:05:34.038030 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.038011    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"
Oct 30 19:05:34.103459 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.103418    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"]
Oct 30 19:05:34.147055 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.147019    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:05:34.147315 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.147115    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:05:34.184650 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.184595    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h75d\" (UniqueName: \"kubernetes.io/projected/b1c6362f-0454-42f1-87fd-492422802d4a-kube-api-access-2h75d\") pod \"maas-api-revoke-sub-2f170e48-s5p2v\" (UID: \"b1c6362f-0454-42f1-87fd-492422802d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"
Oct 30 19:05:34.184837 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.184750    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f170e48-s5p2v\" (UID: \"b1c6362f-0454-42f1-87fd-492422802d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"
Oct 30 19:05:34.185084 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.185066    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f170e48-s5p2v\" (UID: \"b1c6362f-0454-42f1-87fd-492422802d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"
Oct 30 19:05:34.286900 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.286800    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h75d\" (UniqueName: \"kubernetes.io/projected/b1c6362f-0454-42f1-87fd-492422802d4a-kube-api-access-2h75d\") pod \"maas-api-revoke-sub-2f170e48-s5p2v\" (UID: \"b1c6362f-0454-42f1-87fd-492422802d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"
Oct 30 19:05:34.287094 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.286900    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f170e48-s5p2v\" (UID: \"b1c6362f-0454-42f1-87fd-492422802d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"
Oct 30 19:05:34.287227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.287188    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f170e48-s5p2v\" (UID: \"b1c6362f-0454-42f1-87fd-492422802d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"
Oct 30 19:05:34.287877 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.287822    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f170e48-s5p2v\" (UID: \"b1c6362f-0454-42f1-87fd-492422802d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"
Oct 30 19:05:34.289575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.289553    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f170e48-s5p2v\" (UID: \"b1c6362f-0454-42f1-87fd-492422802d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"
Oct 30 19:05:34.343807 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.343777    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h75d\" (UniqueName: \"kubernetes.io/projected/b1c6362f-0454-42f1-87fd-492422802d4a-kube-api-access-2h75d\") pod \"maas-api-revoke-sub-2f170e48-s5p2v\" (UID: \"b1c6362f-0454-42f1-87fd-492422802d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"
Oct 30 19:05:34.382050 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.382015    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"
Oct 30 19:05:34.389500 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.389473    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:05:34.389594 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.389535    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" podUID="aacb99a2-5355-4e64-86bb-ad338415035f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 19:05:34.777789 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:34.777759    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"]
Oct 30 19:05:34.780093 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:05:34.780059    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c6362f_0454_42f1_87fd_492422802d4a.slice/crio-b051097762130e26d6a8f5fdc4d3736d3d40a0cfb90dea19fd069a2d422bd0c0 WatchSource:0}: Error finding container b051097762130e26d6a8f5fdc4d3736d3d40a0cfb90dea19fd069a2d422bd0c0: Status 404 returned error can't find the container with id b051097762130e26d6a8f5fdc4d3736d3d40a0cfb90dea19fd069a2d422bd0c0
Oct 30 19:05:35.133642 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:35.133606    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v" event={"ID":"b1c6362f-0454-42f1-87fd-492422802d4a","Type":"ContainerStarted","Data":"b051097762130e26d6a8f5fdc4d3736d3d40a0cfb90dea19fd069a2d422bd0c0"}
Oct 30 19:05:35.393223 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:35.392488    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:05:35.393223 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:35.392548    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" podUID="aacb99a2-5355-4e64-86bb-ad338415035f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 19:05:36.127651 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.127624    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:05:36.142346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.141602    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-kube-api-access-ncgk8\" (UniqueName: \"kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-kube-api-access-ncgk8\") pod \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") "
Oct 30 19:05:36.142346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.141714    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-socket\") pod \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") "
Oct 30 19:05:36.142346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.141783    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istiod-ca-cert\") pod \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") "
Oct 30 19:05:36.142346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.141854    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-certs\") pod \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") "
Oct 30 19:05:36.142346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.141881    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-data\") pod \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") "
Oct 30 19:05:36.142346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.142052    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-socket" pod "b61093c9-aed4-4f32-85e7-d17180ef7eaf" (UID: "b61093c9-aed4-4f32-85e7-d17180ef7eaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:36.142346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.142118    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-certs" pod "b61093c9-aed4-4f32-85e7-d17180ef7eaf" (UID: "b61093c9-aed4-4f32-85e7-d17180ef7eaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:36.142346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.142131    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-credential-socket\") pod \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") "
Oct 30 19:05:36.142346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.142299    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-token\" (UniqueName: \"kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-token\") pod \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") "
Oct 30 19:05:36.142346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.142289    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-data" pod "b61093c9-aed4-4f32-85e7-d17180ef7eaf" (UID: "b61093c9-aed4-4f32-85e7-d17180ef7eaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:36.142346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.142349    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-envoy\") pod \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") "
Oct 30 19:05:36.142346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.142298    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istiod-ca-cert" pod "b61093c9-aed4-4f32-85e7-d17180ef7eaf" (UID: "b61093c9-aed4-4f32-85e7-d17180ef7eaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:36.142346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.142389    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-podinfo\") pod \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\" (UID: \"b61093c9-aed4-4f32-85e7-d17180ef7eaf\") "
Oct 30 19:05:36.143347 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.142841    6014 generic.go:356] "Generic (PLEG): container finished" podID="b1c6362f-0454-42f1-87fd-492422802d4a" containerID="0254be1caba1bb05bc4c7ac0baf2ac6d2956ba7c1ad7570082589fa3d9723482" exitCode=0
Oct 30 19:05:36.143347 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.143034    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v" event={"ID":"b1c6362f-0454-42f1-87fd-492422802d4a","Type":"ContainerDied","Data":"0254be1caba1bb05bc4c7ac0baf2ac6d2956ba7c1ad7570082589fa3d9723482"}
Oct 30 19:05:36.143347 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.143118    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:36.143347 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.143152    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:36.143347 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.143166    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:36.143347 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.143178    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:36.143347 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.143248    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-credential-socket" pod "b61093c9-aed4-4f32-85e7-d17180ef7eaf" (UID: "b61093c9-aed4-4f32-85e7-d17180ef7eaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:36.144383 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.144358    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-kube-api-access-ncgk8" pod "b61093c9-aed4-4f32-85e7-d17180ef7eaf" (UID: "b61093c9-aed4-4f32-85e7-d17180ef7eaf"). InnerVolumeSpecName "kube-api-access-ncgk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:36.145116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.145087    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-token" pod "b61093c9-aed4-4f32-85e7-d17180ef7eaf" (UID: "b61093c9-aed4-4f32-85e7-d17180ef7eaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:36.145255 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.145235    6014 generic.go:356] "Generic (PLEG): container finished" podID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerID="bd810839730995db5de0fd221120a5bd4719528a44e150ecd3d7520d89b70480" exitCode=0
Oct 30 19:05:36.145314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.145274    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"
Oct 30 19:05:36.145410 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.145389    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw" event={"ID":"b61093c9-aed4-4f32-85e7-d17180ef7eaf","Type":"ContainerDied","Data":"bd810839730995db5de0fd221120a5bd4719528a44e150ecd3d7520d89b70480"}
Oct 30 19:05:36.145466 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.145445    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw" event={"ID":"b61093c9-aed4-4f32-85e7-d17180ef7eaf","Type":"ContainerDied","Data":"a2bf847d28f5b2b2de767ced1213defef026bc6eae8343c88fdfccfbfc5bc970"}
Oct 30 19:05:36.145513 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.145463    6014 scope.go:122] "RemoveContainer" containerID="bd810839730995db5de0fd221120a5bd4719528a44e150ecd3d7520d89b70480"
Oct 30 19:05:36.145789 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.145766    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-envoy" pod "b61093c9-aed4-4f32-85e7-d17180ef7eaf" (UID: "b61093c9-aed4-4f32-85e7-d17180ef7eaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:36.146110 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.146092    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-podinfo" pod "b61093c9-aed4-4f32-85e7-d17180ef7eaf" (UID: "b61093c9-aed4-4f32-85e7-d17180ef7eaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:05:36.282769 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.282714    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:36.282769 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.282757    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:36.282769 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.282774    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:36.283074 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.282791    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61093c9-aed4-4f32-85e7-d17180ef7eaf-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:36.283074 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.282805    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncgk8\" (UniqueName: \"kubernetes.io/projected/b61093c9-aed4-4f32-85e7-d17180ef7eaf-kube-api-access-ncgk8\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:36.285344 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.285321    6014 scope.go:122] "RemoveContainer" containerID="028a9cf53443e6374c8cc5ef56cd7ef5eff5d11b71353449f535d44ccf96d559"
Oct 30 19:05:36.406544 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.406177    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:05:36.406844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.406786    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" podUID="aacb99a2-5355-4e64-86bb-ad338415035f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 19:05:36.496812 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.486064    6014 scope.go:122] "RemoveContainer" containerID="bd810839730995db5de0fd221120a5bd4719528a44e150ecd3d7520d89b70480"
Oct 30 19:05:36.501856 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:05:36.501822    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd810839730995db5de0fd221120a5bd4719528a44e150ecd3d7520d89b70480\": container with ID starting with bd810839730995db5de0fd221120a5bd4719528a44e150ecd3d7520d89b70480 not found: ID does not exist" containerID="bd810839730995db5de0fd221120a5bd4719528a44e150ecd3d7520d89b70480"
Oct 30 19:05:36.502115 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.502086    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd810839730995db5de0fd221120a5bd4719528a44e150ecd3d7520d89b70480"} err="failed to get container status \"bd810839730995db5de0fd221120a5bd4719528a44e150ecd3d7520d89b70480\": rpc error: code = NotFound desc = could not find container \"bd810839730995db5de0fd221120a5bd4719528a44e150ecd3d7520d89b70480\": container with ID starting with bd810839730995db5de0fd221120a5bd4719528a44e150ecd3d7520d89b70480 not found: ID does not exist"
Oct 30 19:05:36.502292 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.502279    6014 scope.go:122] "RemoveContainer" containerID="028a9cf53443e6374c8cc5ef56cd7ef5eff5d11b71353449f535d44ccf96d559"
Oct 30 19:05:36.502937 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:05:36.502885    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028a9cf53443e6374c8cc5ef56cd7ef5eff5d11b71353449f535d44ccf96d559\": container with ID starting with 028a9cf53443e6374c8cc5ef56cd7ef5eff5d11b71353449f535d44ccf96d559 not found: ID does not exist" containerID="028a9cf53443e6374c8cc5ef56cd7ef5eff5d11b71353449f535d44ccf96d559"
Oct 30 19:05:36.503121 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.502917    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028a9cf53443e6374c8cc5ef56cd7ef5eff5d11b71353449f535d44ccf96d559"} err="failed to get container status \"028a9cf53443e6374c8cc5ef56cd7ef5eff5d11b71353449f535d44ccf96d559\": rpc error: code = NotFound desc = could not find container \"028a9cf53443e6374c8cc5ef56cd7ef5eff5d11b71353449f535d44ccf96d559\": container with ID starting with 028a9cf53443e6374c8cc5ef56cd7ef5eff5d11b71353449f535d44ccf96d559 not found: ID does not exist"
Oct 30 19:05:36.690543 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.690467    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"]
Oct 30 19:05:36.694755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:36.694732    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-cd57e7dd-openshift-default-7db5477bfc-qb5xw"]
Oct 30 19:05:37.392202 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:37.391263    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:05:37.392202 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:37.391332    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" podUID="aacb99a2-5355-4e64-86bb-ad338415035f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 19:05:37.524623 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:37.523895    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" path="/var/lib/kubelet/pods/b61093c9-aed4-4f32-85e7-d17180ef7eaf/volumes"
Oct 30 19:05:37.572164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:37.570217    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"
Oct 30 19:05:37.631620 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:37.631581    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c6362f-0454-42f1-87fd-492422802d4a-kube-api-access-2h75d\" (UniqueName: \"kubernetes.io/projected/b1c6362f-0454-42f1-87fd-492422802d4a-kube-api-access-2h75d\") pod \"b1c6362f-0454-42f1-87fd-492422802d4a\" (UID: \"b1c6362f-0454-42f1-87fd-492422802d4a\") "
Oct 30 19:05:37.631998 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:37.631745    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-xks-ca\") pod \"b1c6362f-0454-42f1-87fd-492422802d4a\" (UID: \"b1c6362f-0454-42f1-87fd-492422802d4a\") "
Oct 30 19:05:37.631998 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:37.631821    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-openshift-ca\") pod \"b1c6362f-0454-42f1-87fd-492422802d4a\" (UID: \"b1c6362f-0454-42f1-87fd-492422802d4a\") "
Oct 30 19:05:37.634207 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:37.634179    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-openshift-ca" pod "b1c6362f-0454-42f1-87fd-492422802d4a" (UID: "b1c6362f-0454-42f1-87fd-492422802d4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:37.639517 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:37.639489    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c6362f-0454-42f1-87fd-492422802d4a-kube-api-access-2h75d" pod "b1c6362f-0454-42f1-87fd-492422802d4a" (UID: "b1c6362f-0454-42f1-87fd-492422802d4a"). InnerVolumeSpecName "kube-api-access-2h75d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:37.641276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:37.641256    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-xks-ca" pod "b1c6362f-0454-42f1-87fd-492422802d4a" (UID: "b1c6362f-0454-42f1-87fd-492422802d4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:37.734954 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:37.734915    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h75d\" (UniqueName: \"kubernetes.io/projected/b1c6362f-0454-42f1-87fd-492422802d4a-kube-api-access-2h75d\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:37.734954 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:37.734957    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:37.735371 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:37.734973    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c6362f-0454-42f1-87fd-492422802d4a-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:38.196825 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:38.196796    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"
Oct 30 19:05:38.197019 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:38.196836    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v" event={"ID":"b1c6362f-0454-42f1-87fd-492422802d4a","Type":"ContainerDied","Data":"b051097762130e26d6a8f5fdc4d3736d3d40a0cfb90dea19fd069a2d422bd0c0"}
Oct 30 19:05:38.197019 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:38.196893    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b051097762130e26d6a8f5fdc4d3736d3d40a0cfb90dea19fd069a2d422bd0c0"
Oct 30 19:05:38.389502 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:38.389468    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:05:38.389694 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:38.389533    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" podUID="aacb99a2-5355-4e64-86bb-ad338415035f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 19:05:39.389346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:39.389300    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:05:39.389848 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:39.389376    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" podUID="aacb99a2-5355-4e64-86bb-ad338415035f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 19:05:40.837877 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:40.837845    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:40.838446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:40.838399    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:40.839537 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:40.839510    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57"
Oct 30 19:05:41.048863 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:41.048835    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"
Oct 30 19:05:41.061737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:41.061707    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"]
Oct 30 19:05:41.065525 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:41.065503    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9403a0ab-9nmlc"]
Oct 30 19:05:41.477908 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:41.477876    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="272737d7-b2ef-4706-8cb7-bf089905b287" path="/var/lib/kubelet/pods/272737d7-b2ef-4706-8cb7-bf089905b287/volumes"
Oct 30 19:05:53.260381 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:05:53.260339    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:05:57.051287 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:57.051253    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"]
Oct 30 19:05:57.100022 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:57.099991    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b4ba65-ldqz2"]
Oct 30 19:05:57.481689 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:05:57.481619    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f582f52-06a8-4f9e-806d-06bf591b64e6" path="/var/lib/kubelet/pods/3f582f52-06a8-4f9e-806d-06bf591b64e6/volumes"
Oct 30 19:06:05.904793 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:05.904753    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:06:06.049766 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:06.049726    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"]
Oct 30 19:06:06.053906 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:06.053874    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b839a2d9-58p2k"]
Oct 30 19:06:07.478474 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:07.478444    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccae37a5-4937-41df-9cae-6f0a76a40b38" path="/var/lib/kubelet/pods/ccae37a5-4937-41df-9cae-6f0a76a40b38/volumes"
Oct 30 19:06:21.567822 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.567774    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"]
Oct 30 19:06:21.569104 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.569074    6014 cpu_manager.go:414] "containerMap: removing container" podUID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerName="istio-proxy"
Oct 30 19:06:21.569104 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.569105    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerName="istio-proxy"
Oct 30 19:06:21.569406 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.569123    6014 cpu_manager.go:414] "containerMap: removing container" podUID="b1c6362f-0454-42f1-87fd-492422802d4a" containerName="revoke-keys"
Oct 30 19:06:21.569406 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.569131    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c6362f-0454-42f1-87fd-492422802d4a" containerName="revoke-keys"
Oct 30 19:06:21.569406 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.569225    6014 cpu_manager.go:414] "containerMap: removing container" podUID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerName="istio-proxy"
Oct 30 19:06:21.569406 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.569254    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerName="istio-proxy"
Oct 30 19:06:21.569792 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.569426    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerName="istio-proxy"
Oct 30 19:06:21.569792 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.569447    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61093c9-aed4-4f32-85e7-d17180ef7eaf" containerName="istio-proxy"
Oct 30 19:06:21.569792 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.569457    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c6362f-0454-42f1-87fd-492422802d4a" containerName="revoke-keys"
Oct 30 19:06:21.574408 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.574303    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"
Oct 30 19:06:21.626603 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.626567    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"]
Oct 30 19:06:21.650962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.650919    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9a80a2-6c53-40a3-a5a3-44257e705082-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4\" (UID: \"1b9a80a2-6c53-40a3-a5a3-44257e705082\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"
Oct 30 19:06:21.651538 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.651508    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8x2j\" (UniqueName: \"kubernetes.io/projected/1b9a80a2-6c53-40a3-a5a3-44257e705082-kube-api-access-c8x2j\") pod \"payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4\" (UID: \"1b9a80a2-6c53-40a3-a5a3-44257e705082\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"
Oct 30 19:06:21.672228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.670416    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-732a3f\""
Oct 30 19:06:21.672228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.670588    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-732a3f-dockercfg-sbd79\""
Oct 30 19:06:21.757159 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.757101    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9a80a2-6c53-40a3-a5a3-44257e705082-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4\" (UID: \"1b9a80a2-6c53-40a3-a5a3-44257e705082\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"
Oct 30 19:06:21.757599 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.757579    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8x2j\" (UniqueName: \"kubernetes.io/projected/1b9a80a2-6c53-40a3-a5a3-44257e705082-kube-api-access-c8x2j\") pod \"payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4\" (UID: \"1b9a80a2-6c53-40a3-a5a3-44257e705082\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"
Oct 30 19:06:21.760844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.760570    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9a80a2-6c53-40a3-a5a3-44257e705082-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4\" (UID: \"1b9a80a2-6c53-40a3-a5a3-44257e705082\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"
Oct 30 19:06:21.817173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.817114    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8x2j\" (UniqueName: \"kubernetes.io/projected/1b9a80a2-6c53-40a3-a5a3-44257e705082-kube-api-access-c8x2j\") pod \"payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4\" (UID: \"1b9a80a2-6c53-40a3-a5a3-44257e705082\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"
Oct 30 19:06:21.924919 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:21.924831    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"
Oct 30 19:06:22.357296 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:22.357271    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"]
Oct 30 19:06:22.358929 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:06:22.358901    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9a80a2_6c53_40a3_a5a3_44257e705082.slice/crio-341f2ef2354a417b1d20e39e21879ddc6f9f02053e71d1a7782b2611d46faca3 WatchSource:0}: Error finding container 341f2ef2354a417b1d20e39e21879ddc6f9f02053e71d1a7782b2611d46faca3: Status 404 returned error can't find the container with id 341f2ef2354a417b1d20e39e21879ddc6f9f02053e71d1a7782b2611d46faca3
Oct 30 19:06:22.673420 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:22.673383    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4" event={"ID":"1b9a80a2-6c53-40a3-a5a3-44257e705082","Type":"ContainerStarted","Data":"5ecd9f25b19dc016cb76d7f36dff3d8a6314842c1461bf7bd5443714674746fa"}
Oct 30 19:06:22.673947 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:22.673449    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4" event={"ID":"1b9a80a2-6c53-40a3-a5a3-44257e705082","Type":"ContainerStarted","Data":"341f2ef2354a417b1d20e39e21879ddc6f9f02053e71d1a7782b2611d46faca3"}
Oct 30 19:06:22.673947 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:22.673508    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"
Oct 30 19:06:22.793212 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:22.793130    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4" podStartSLOduration=1.7931155300000001 podStartE2EDuration="1.79311553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:22.788714793 +0000 UTC m=+1621.431882407" watchObservedRunningTime="2026-10-30 19:06:22.79311553 +0000 UTC m=+1621.436283143"
Oct 30 19:06:23.265825 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:06:23.265786    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:06:23.593605 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:23.593573    6014 scope.go:122] "RemoveContainer" containerID="2396cee5c6e9abae742469b89e050a4e89bdffb7f4de99cd60c6d051d658e9d0"
Oct 30 19:06:25.123513 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:25.123476    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"]
Oct 30 19:06:25.202602 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:25.202564    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedce872-wjtb5"]
Oct 30 19:06:25.478268 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:25.478194    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9561525e-428b-4b15-8de4-9968b36566cb" path="/var/lib/kubelet/pods/9561525e-428b-4b15-8de4-9968b36566cb/volumes"
Oct 30 19:06:29.921602 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:29.921570    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"]
Oct 30 19:06:29.927922 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:29.927901    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:29.983166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:29.982285    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ae6a138d-openshift-default-dockercfg-hrsvc\""
Oct 30 19:06:30.062786 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.062746    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.063001 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.062799    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.063001 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.062864    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.063001 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.062915    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1c4684-874c-4d3d-aebb-517a892cb376-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.063219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.063125    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5snz\" (UniqueName: \"kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-kube-api-access-r5snz\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.063219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.063190    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.063335 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.063236    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.063335 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.063275    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.063459 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.063342    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.067663 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.067624    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"]
Oct 30 19:06:30.164736 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.164699    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1c4684-874c-4d3d-aebb-517a892cb376-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.164932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.164805    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5snz\" (UniqueName: \"kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-kube-api-access-r5snz\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.164932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.164839    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.164932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.164864    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.164932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.164898    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.164932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.164930    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.165253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.165003    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.165253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.165032    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.165253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.165108    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.165401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.165328    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.165401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.165339    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.165537 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.165512    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.165610 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.165520    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.165894 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.165872    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1c4684-874c-4d3d-aebb-517a892cb376-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.167763 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.167741    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.167960 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.167940    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.231554 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.231483    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.233283 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.233265    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5snz\" (UniqueName: \"kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-kube-api-access-r5snz\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.243288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.243261    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:30.735852 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.735811    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"]
Oct 30 19:06:30.736559 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:06:30.736533    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1c4684_874c_4d3d_aebb_517a892cb376.slice/crio-f601df9a9853624aa0c36e65a49c8ae29df1f10d6bd4016f253d46f4e3696ee8 WatchSource:0}: Error finding container f601df9a9853624aa0c36e65a49c8ae29df1f10d6bd4016f253d46f4e3696ee8: Status 404 returned error can't find the container with id f601df9a9853624aa0c36e65a49c8ae29df1f10d6bd4016f253d46f4e3696ee8
Oct 30 19:06:30.739524 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.739488    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:30.739638 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.739564    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:30.739638 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.739591    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:30.786428 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:30.783965    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" event={"ID":"6d1c4684-874c-4d3d-aebb-517a892cb376","Type":"ContainerStarted","Data":"f601df9a9853624aa0c36e65a49c8ae29df1f10d6bd4016f253d46f4e3696ee8"}
Oct 30 19:06:31.794244 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:31.794194    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" event={"ID":"6d1c4684-874c-4d3d-aebb-517a892cb376","Type":"ContainerStarted","Data":"2bbfb018dd3c8ce4fa048f873c3971203c5acac2502e45a81e7772984231cbe2"}
Oct 30 19:06:31.957905 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:31.957847    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" podStartSLOduration=2.957832329 podStartE2EDuration="2.957832329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:31.954541212 +0000 UTC m=+1630.597708825" watchObservedRunningTime="2026-10-30 19:06:31.957832329 +0000 UTC m=+1630.600999942"
Oct 30 19:06:32.243752 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:32.243720    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:32.249718 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:32.249692    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:32.805803 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:32.805669    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:32.807081 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:32.807058    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:06:33.685196 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:33.685169    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"
Oct 30 19:06:37.478694 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:37.478663    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"]
Oct 30 19:06:37.479107 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:37.478874    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng" podUID="2b35af62-6e06-4f5e-8c63-4b8a479f6845" containerName="istio-proxy" containerID="cri-o://4ec00da1183b450ccb3e4a43f23b5aa198a22d5988a0daade0aaeffacf39c377" gracePeriod=30
Oct 30 19:06:37.521031 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:37.520996    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"]
Oct 30 19:06:37.521313 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:37.521283    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59" podUID="0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" containerName="istio-proxy" containerID="cri-o://7502cc71ab581d311f1d876ff8a9d2e8d2d19dc4333c5088eeb192985480b7db" gracePeriod=30
Oct 30 19:06:37.527652 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:37.527618    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"]
Oct 30 19:06:37.527852 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:37.527832    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" podUID="43fd9a91-35f5-454e-95ad-8f2c446ed639" containerName="istio-proxy" containerID="cri-o://9c764953ac23ce7d8eda0fb182c49f72043b838338cf62b3f29526310a7fc221" gracePeriod=30
Oct 30 19:06:37.569507 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:37.569465    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"]
Oct 30 19:06:37.569769 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:37.569726    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" podUID="9f961678-957e-4e77-94c2-c551cb6627cb" containerName="istio-proxy" containerID="cri-o://9c59324f3efe53422e15b8633234278cb257be52251de28b0f57013d027564d1" gracePeriod=30
Oct 30 19:06:39.420869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:39.420833    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"]
Oct 30 19:06:39.426012 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:39.425995    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"
Oct 30 19:06:39.486596 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:39.486567    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"]
Oct 30 19:06:39.529857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:39.529828    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ae6a138d\""
Oct 30 19:06:39.530087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:39.529840    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ae6a138d-dockercfg-mxcpd\""
Oct 30 19:06:39.535105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:39.535072    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8298r\" (UniqueName: \"kubernetes.io/projected/75587314-c5fa-4ce1-8a13-dced50dde605-kube-api-access-8298r\") pod \"payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv\" (UID: \"75587314-c5fa-4ce1-8a13-dced50dde605\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"
Oct 30 19:06:39.536106 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:39.536075    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75587314-c5fa-4ce1-8a13-dced50dde605-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv\" (UID: \"75587314-c5fa-4ce1-8a13-dced50dde605\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"
Oct 30 19:06:39.665219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:39.655881    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75587314-c5fa-4ce1-8a13-dced50dde605-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv\" (UID: \"75587314-c5fa-4ce1-8a13-dced50dde605\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"
Oct 30 19:06:39.665219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:39.656208    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8298r\" (UniqueName: \"kubernetes.io/projected/75587314-c5fa-4ce1-8a13-dced50dde605-kube-api-access-8298r\") pod \"payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv\" (UID: \"75587314-c5fa-4ce1-8a13-dced50dde605\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"
Oct 30 19:06:39.665219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:39.657684    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75587314-c5fa-4ce1-8a13-dced50dde605-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv\" (UID: \"75587314-c5fa-4ce1-8a13-dced50dde605\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"
Oct 30 19:06:39.749357 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:39.745652    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8298r\" (UniqueName: \"kubernetes.io/projected/75587314-c5fa-4ce1-8a13-dced50dde605-kube-api-access-8298r\") pod \"payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv\" (UID: \"75587314-c5fa-4ce1-8a13-dced50dde605\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"
Oct 30 19:06:39.749357 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:39.746021    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"
Oct 30 19:06:40.198770 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:40.198735    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"]
Oct 30 19:06:40.198910 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:06:40.198866    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75587314_c5fa_4ce1_8a13_dced50dde605.slice/crio-b205a664dc8cff4dc12b33085e4f8f3b6bbf6a93bd22e0aa474057d960f8332a WatchSource:0}: Error finding container b205a664dc8cff4dc12b33085e4f8f3b6bbf6a93bd22e0aa474057d960f8332a: Status 404 returned error can't find the container with id b205a664dc8cff4dc12b33085e4f8f3b6bbf6a93bd22e0aa474057d960f8332a
Oct 30 19:06:40.218756 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:40.218715    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:06:40.218918 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:40.218787    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng" podUID="2b35af62-6e06-4f5e-8c63-4b8a479f6845" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:40.932731 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:40.932697    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv" event={"ID":"75587314-c5fa-4ce1-8a13-dced50dde605","Type":"ContainerStarted","Data":"5280ac327f9af4c744cfd9c0cb95a318c4a14288e0a0808672975054779de451"}
Oct 30 19:06:40.933241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:40.932760    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv" event={"ID":"75587314-c5fa-4ce1-8a13-dced50dde605","Type":"ContainerStarted","Data":"b205a664dc8cff4dc12b33085e4f8f3b6bbf6a93bd22e0aa474057d960f8332a"}
Oct 30 19:06:40.933241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:40.932792    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"
Oct 30 19:06:41.049904 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:41.049842    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv" podStartSLOduration=2.049826759 podStartE2EDuration="2.049826759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:41.045866867 +0000 UTC m=+1639.689034483" watchObservedRunningTime="2026-10-30 19:06:41.049826759 +0000 UTC m=+1639.692994393"
Oct 30 19:06:41.280801 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:41.280716    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:06:41.280801 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:41.280794    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" podUID="9f961678-957e-4e77-94c2-c551cb6627cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:41.303905 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:41.303705    6014 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:06:41.303905 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:41.303774    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" podUID="43fd9a91-35f5-454e-95ad-8f2c446ed639" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:42.061212 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.060603    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"]
Oct 30 19:06:42.119651 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.118592    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7812ebe-v2b2h"]
Oct 30 19:06:42.862248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.862228    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 19:06:42.953100 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.952775    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-socket\") pod \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") "
Oct 30 19:06:42.953100 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.952828    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-data\") pod \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") "
Oct 30 19:06:42.953100 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.952924    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istiod-ca-cert\") pod \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") "
Oct 30 19:06:42.953100 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.953095    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-envoy\") pod \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") "
Oct 30 19:06:42.957340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.953198    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-socket" pod "2b35af62-6e06-4f5e-8c63-4b8a479f6845" (UID: "2b35af62-6e06-4f5e-8c63-4b8a479f6845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.957340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.953207    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-kube-api-access-6k29s\" (UniqueName: \"kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-kube-api-access-6k29s\") pod \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") "
Oct 30 19:06:42.957340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.953511    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-certs\") pod \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") "
Oct 30 19:06:42.957340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.954787    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istiod-ca-cert" pod "2b35af62-6e06-4f5e-8c63-4b8a479f6845" (UID: "2b35af62-6e06-4f5e-8c63-4b8a479f6845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:42.957340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.955033    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-data" pod "2b35af62-6e06-4f5e-8c63-4b8a479f6845" (UID: "2b35af62-6e06-4f5e-8c63-4b8a479f6845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.957340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.955641    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-credential-socket\") pod \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") "
Oct 30 19:06:42.957340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.955695    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-podinfo\") pod \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") "
Oct 30 19:06:42.957340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.955757    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-token\" (UniqueName: \"kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-token\") pod \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\" (UID: \"2b35af62-6e06-4f5e-8c63-4b8a479f6845\") "
Oct 30 19:06:42.957340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.956704    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.957340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.956726    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.957340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.956741    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.957984 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.957597    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-credential-socket" pod "2b35af62-6e06-4f5e-8c63-4b8a479f6845" (UID: "2b35af62-6e06-4f5e-8c63-4b8a479f6845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.957984 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.957830    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-certs" pod "2b35af62-6e06-4f5e-8c63-4b8a479f6845" (UID: "2b35af62-6e06-4f5e-8c63-4b8a479f6845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.960366 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.960270    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-envoy" pod "2b35af62-6e06-4f5e-8c63-4b8a479f6845" (UID: "2b35af62-6e06-4f5e-8c63-4b8a479f6845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.960366 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.960331    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-kube-api-access-6k29s" pod "2b35af62-6e06-4f5e-8c63-4b8a479f6845" (UID: "2b35af62-6e06-4f5e-8c63-4b8a479f6845"). InnerVolumeSpecName "kube-api-access-6k29s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:42.976159 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.964458    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-token" pod "2b35af62-6e06-4f5e-8c63-4b8a479f6845" (UID: "2b35af62-6e06-4f5e-8c63-4b8a479f6845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:42.980270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.980234    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-podinfo" pod "2b35af62-6e06-4f5e-8c63-4b8a479f6845" (UID: "2b35af62-6e06-4f5e-8c63-4b8a479f6845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:06:42.981531 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.980945    6014 generic.go:356] "Generic (PLEG): container finished" podID="43fd9a91-35f5-454e-95ad-8f2c446ed639" containerID="9c764953ac23ce7d8eda0fb182c49f72043b838338cf62b3f29526310a7fc221" exitCode=0
Oct 30 19:06:42.981531 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.981180    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" event={"ID":"43fd9a91-35f5-454e-95ad-8f2c446ed639","Type":"ContainerDied","Data":"9c764953ac23ce7d8eda0fb182c49f72043b838338cf62b3f29526310a7fc221"}
Oct 30 19:06:42.984059 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.984035    6014 generic.go:356] "Generic (PLEG): container finished" podID="9f961678-957e-4e77-94c2-c551cb6627cb" containerID="9c59324f3efe53422e15b8633234278cb257be52251de28b0f57013d027564d1" exitCode=0
Oct 30 19:06:42.984217 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.984106    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" event={"ID":"9f961678-957e-4e77-94c2-c551cb6627cb","Type":"ContainerDied","Data":"9c59324f3efe53422e15b8633234278cb257be52251de28b0f57013d027564d1"}
Oct 30 19:06:42.988048 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.988024    6014 generic.go:356] "Generic (PLEG): container finished" podID="0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" containerID="7502cc71ab581d311f1d876ff8a9d2e8d2d19dc4333c5088eeb192985480b7db" exitCode=0
Oct 30 19:06:42.988210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.988105    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59" event={"ID":"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c","Type":"ContainerDied","Data":"7502cc71ab581d311f1d876ff8a9d2e8d2d19dc4333c5088eeb192985480b7db"}
Oct 30 19:06:42.988279 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.988260    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59" event={"ID":"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c","Type":"ContainerDied","Data":"4efc25277db990acc91affe286d5f4858e5216bc1ee256e1ae836520a655e030"}
Oct 30 19:06:42.988340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.988302    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4efc25277db990acc91affe286d5f4858e5216bc1ee256e1ae836520a655e030"
Oct 30 19:06:42.990262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.990222    6014 generic.go:356] "Generic (PLEG): container finished" podID="2b35af62-6e06-4f5e-8c63-4b8a479f6845" containerID="4ec00da1183b450ccb3e4a43f23b5aa198a22d5988a0daade0aaeffacf39c377" exitCode=0
Oct 30 19:06:42.990379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.990253    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng" event={"ID":"2b35af62-6e06-4f5e-8c63-4b8a479f6845","Type":"ContainerDied","Data":"4ec00da1183b450ccb3e4a43f23b5aa198a22d5988a0daade0aaeffacf39c377"}
Oct 30 19:06:42.990379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.990312    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng" event={"ID":"2b35af62-6e06-4f5e-8c63-4b8a479f6845","Type":"ContainerDied","Data":"058efba49120a3f17729e7cffedbbab1c104ebcaebca61d951a6b2be10d74e81"}
Oct 30 19:06:42.990379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.990353    6014 scope.go:122] "RemoveContainer" containerID="4ec00da1183b450ccb3e4a43f23b5aa198a22d5988a0daade0aaeffacf39c377"
Oct 30 19:06:42.990379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.990371    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"
Oct 30 19:06:42.992999 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:42.992978    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:06:43.005195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.005098    6014 scope.go:122] "RemoveContainer" containerID="4ec00da1183b450ccb3e4a43f23b5aa198a22d5988a0daade0aaeffacf39c377"
Oct 30 19:06:43.005427 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.005403    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 19:06:43.005587 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:06:43.005564    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec00da1183b450ccb3e4a43f23b5aa198a22d5988a0daade0aaeffacf39c377\": container with ID starting with 4ec00da1183b450ccb3e4a43f23b5aa198a22d5988a0daade0aaeffacf39c377 not found: ID does not exist" containerID="4ec00da1183b450ccb3e4a43f23b5aa198a22d5988a0daade0aaeffacf39c377"
Oct 30 19:06:43.005640 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.005601    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec00da1183b450ccb3e4a43f23b5aa198a22d5988a0daade0aaeffacf39c377"} err="failed to get container status \"4ec00da1183b450ccb3e4a43f23b5aa198a22d5988a0daade0aaeffacf39c377\": rpc error: code = NotFound desc = could not find container \"4ec00da1183b450ccb3e4a43f23b5aa198a22d5988a0daade0aaeffacf39c377\": container with ID starting with 4ec00da1183b450ccb3e4a43f23b5aa198a22d5988a0daade0aaeffacf39c377 not found: ID does not exist"
Oct 30 19:06:43.010044 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.010025    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 19:06:43.048867 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.048773    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"]
Oct 30 19:06:43.053379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.053351    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34c04623-cqdtw"]
Oct 30 19:06:43.057905 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.057871    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istiod-ca-cert\") pod \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") "
Oct 30 19:06:43.058043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.057954    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-socket\") pod \"9f961678-957e-4e77-94c2-c551cb6627cb\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") "
Oct 30 19:06:43.058111 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058042    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-certs\") pod \"43fd9a91-35f5-454e-95ad-8f2c446ed639\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") "
Oct 30 19:06:43.058111 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058094    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-socket\") pod \"43fd9a91-35f5-454e-95ad-8f2c446ed639\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") "
Oct 30 19:06:43.058235 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058122    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-envoy\") pod \"9f961678-957e-4e77-94c2-c551cb6627cb\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") "
Oct 30 19:06:43.058235 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058178    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-socket" pod "9f961678-957e-4e77-94c2-c551cb6627cb" (UID: "9f961678-957e-4e77-94c2-c551cb6627cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.058333 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058236    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-token\" (UniqueName: \"kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-token\") pod \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") "
Oct 30 19:06:43.058382 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058327    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istiod-ca-cert" pod "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" (UID: "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:43.058382 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058333    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-certs" pod "43fd9a91-35f5-454e-95ad-8f2c446ed639" (UID: "43fd9a91-35f5-454e-95ad-8f2c446ed639"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.058382 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058360    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f961678-957e-4e77-94c2-c551cb6627cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f961678-957e-4e77-94c2-c551cb6627cb-istiod-ca-cert\") pod \"9f961678-957e-4e77-94c2-c551cb6627cb\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") "
Oct 30 19:06:43.058532 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058400    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-envoy\") pod \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") "
Oct 30 19:06:43.058532 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058413    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-socket" pod "43fd9a91-35f5-454e-95ad-8f2c446ed639" (UID: "43fd9a91-35f5-454e-95ad-8f2c446ed639"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.058532 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058469    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-socket\") pod \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") "
Oct 30 19:06:43.058686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058543    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-data\") pod \"9f961678-957e-4e77-94c2-c551cb6627cb\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") "
Oct 30 19:06:43.058686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058585    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-kube-api-access-jdpgr\" (UniqueName: \"kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-kube-api-access-jdpgr\") pod \"43fd9a91-35f5-454e-95ad-8f2c446ed639\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") "
Oct 30 19:06:43.058686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058617    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-data\") pod \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") "
Oct 30 19:06:43.058686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058669    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-certs\") pod \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") "
Oct 30 19:06:43.058882 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058726    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-credential-socket\") pod \"43fd9a91-35f5-454e-95ad-8f2c446ed639\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") "
Oct 30 19:06:43.058882 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058755    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-podinfo\") pod \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") "
Oct 30 19:06:43.058882 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058800    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-kube-api-access-dqrfr\" (UniqueName: \"kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-kube-api-access-dqrfr\") pod \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") "
Oct 30 19:06:43.058882 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058804    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-data" pod "9f961678-957e-4e77-94c2-c551cb6627cb" (UID: "9f961678-957e-4e77-94c2-c551cb6627cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.058882 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058829    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f961678-957e-4e77-94c2-c551cb6627cb-istiod-ca-cert" pod "9f961678-957e-4e77-94c2-c551cb6627cb" (UID: "9f961678-957e-4e77-94c2-c551cb6627cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:43.058882 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058837    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-token\" (UniqueName: \"kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-token\") pod \"43fd9a91-35f5-454e-95ad-8f2c446ed639\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") "
Oct 30 19:06:43.058882 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058887    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-podinfo\") pod \"43fd9a91-35f5-454e-95ad-8f2c446ed639\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") "
Oct 30 19:06:43.059606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.058971    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43fd9a91-35f5-454e-95ad-8f2c446ed639-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43fd9a91-35f5-454e-95ad-8f2c446ed639-istiod-ca-cert\") pod \"43fd9a91-35f5-454e-95ad-8f2c446ed639\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") "
Oct 30 19:06:43.059606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.059013    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-credential-socket\") pod \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\" (UID: \"0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c\") "
Oct 30 19:06:43.059606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.059044    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f961678-957e-4e77-94c2-c551cb6627cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f961678-957e-4e77-94c2-c551cb6627cb-istio-podinfo\") pod \"9f961678-957e-4e77-94c2-c551cb6627cb\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") "
Oct 30 19:06:43.059606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.059313    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-credential-socket\") pod \"9f961678-957e-4e77-94c2-c551cb6627cb\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") "
Oct 30 19:06:43.059606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.059379    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-kube-api-access-xpbxv\" (UniqueName: \"kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-kube-api-access-xpbxv\") pod \"9f961678-957e-4e77-94c2-c551cb6627cb\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") "
Oct 30 19:06:43.059606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.059458    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-envoy\") pod \"43fd9a91-35f5-454e-95ad-8f2c446ed639\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") "
Oct 30 19:06:43.059606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.059499    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-data\") pod \"43fd9a91-35f5-454e-95ad-8f2c446ed639\" (UID: \"43fd9a91-35f5-454e-95ad-8f2c446ed639\") "
Oct 30 19:06:43.059606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.059567    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-certs\") pod \"9f961678-957e-4e77-94c2-c551cb6627cb\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") "
Oct 30 19:06:43.060054 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.059621    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-istio-token\" (UniqueName: \"kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-istio-token\") pod \"9f961678-957e-4e77-94c2-c551cb6627cb\" (UID: \"9f961678-957e-4e77-94c2-c551cb6627cb\") "
Oct 30 19:06:43.060110 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.060083    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-certs" pod "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" (UID: "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.060633 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.060603    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-socket" pod "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" (UID: "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.061432 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061376    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-envoy" pod "9f961678-957e-4e77-94c2-c551cb6627cb" (UID: "9f961678-957e-4e77-94c2-c551cb6627cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061449    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-token" pod "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" (UID: "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061502    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061719    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f961678-957e-4e77-94c2-c551cb6627cb-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061736    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061751    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061767    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061781    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061796    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k29s\" (UniqueName: \"kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-kube-api-access-6k29s\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061812    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061828    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b35af62-6e06-4f5e-8c63-4b8a479f6845-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061844    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061859    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b35af62-6e06-4f5e-8c63-4b8a479f6845-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061876    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.061886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061892    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.062844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061906    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.062844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061642    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-data" pod "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" (UID: "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.062844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061656    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-credential-socket" pod "9f961678-957e-4e77-94c2-c551cb6627cb" (UID: "9f961678-957e-4e77-94c2-c551cb6627cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.062844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.061958    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-token" pod "43fd9a91-35f5-454e-95ad-8f2c446ed639" (UID: "43fd9a91-35f5-454e-95ad-8f2c446ed639"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:43.062844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.062207    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-credential-socket" pod "43fd9a91-35f5-454e-95ad-8f2c446ed639" (UID: "43fd9a91-35f5-454e-95ad-8f2c446ed639"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.062844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.062416    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-envoy" pod "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" (UID: "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.062844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.062433    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-credential-socket" pod "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" (UID: "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.062844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.062798    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43fd9a91-35f5-454e-95ad-8f2c446ed639-istiod-ca-cert" pod "43fd9a91-35f5-454e-95ad-8f2c446ed639" (UID: "43fd9a91-35f5-454e-95ad-8f2c446ed639"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:43.063297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.062827    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-istio-token" pod "9f961678-957e-4e77-94c2-c551cb6627cb" (UID: "9f961678-957e-4e77-94c2-c551cb6627cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:43.063297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.063109    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-data" pod "43fd9a91-35f5-454e-95ad-8f2c446ed639" (UID: "43fd9a91-35f5-454e-95ad-8f2c446ed639"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.063417 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.063351    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-certs" pod "9f961678-957e-4e77-94c2-c551cb6627cb" (UID: "9f961678-957e-4e77-94c2-c551cb6627cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.063979 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.063838    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-kube-api-access-jdpgr" pod "43fd9a91-35f5-454e-95ad-8f2c446ed639" (UID: "43fd9a91-35f5-454e-95ad-8f2c446ed639"). InnerVolumeSpecName "kube-api-access-jdpgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:43.064271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.064246    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-envoy" pod "43fd9a91-35f5-454e-95ad-8f2c446ed639" (UID: "43fd9a91-35f5-454e-95ad-8f2c446ed639"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:43.064417 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.064398    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-kube-api-access-dqrfr" pod "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" (UID: "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c"). InnerVolumeSpecName "kube-api-access-dqrfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:43.064535 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.064518    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-podinfo" pod "43fd9a91-35f5-454e-95ad-8f2c446ed639" (UID: "43fd9a91-35f5-454e-95ad-8f2c446ed639"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:06:43.064747 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.064720    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f961678-957e-4e77-94c2-c551cb6627cb-istio-podinfo" pod "9f961678-957e-4e77-94c2-c551cb6627cb" (UID: "9f961678-957e-4e77-94c2-c551cb6627cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:06:43.065216 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.065195    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-podinfo" pod "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" (UID: "0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:06:43.065449 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.065430    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-kube-api-access-xpbxv" pod "9f961678-957e-4e77-94c2-c551cb6627cb" (UID: "9f961678-957e-4e77-94c2-c551cb6627cb"). InnerVolumeSpecName "kube-api-access-xpbxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:43.164010 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.163974    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164010 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164005    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpbxv\" (UniqueName: \"kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-kube-api-access-xpbxv\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164010 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164015    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164024    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164035    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164046    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f961678-957e-4e77-94c2-c551cb6627cb-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164055    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f961678-957e-4e77-94c2-c551cb6627cb-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164063    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164071    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164079    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdpgr\" (UniqueName: \"kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-kube-api-access-jdpgr\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164089    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164098    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fd9a91-35f5-454e-95ad-8f2c446ed639-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164107    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164115    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqrfr\" (UniqueName: \"kubernetes.io/projected/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-kube-api-access-dqrfr\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164124    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164132    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43fd9a91-35f5-454e-95ad-8f2c446ed639-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164157    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43fd9a91-35f5-454e-95ad-8f2c446ed639-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164170    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.164270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.164178    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f961678-957e-4e77-94c2-c551cb6627cb-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.336502 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.336468    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"]
Oct 30 19:06:43.341431 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.341397    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-dtlng"]
Oct 30 19:06:43.480130 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.480095    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f26ae2-4b28-4b53-8619-9fab66457ea6" path="/var/lib/kubelet/pods/14f26ae2-4b28-4b53-8619-9fab66457ea6/volumes"
Oct 30 19:06:43.480768 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.480745    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b35af62-6e06-4f5e-8c63-4b8a479f6845" path="/var/lib/kubelet/pods/2b35af62-6e06-4f5e-8c63-4b8a479f6845/volumes"
Oct 30 19:06:43.481317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.481297    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f619c5-b367-4e79-959f-1e022fc48c23" path="/var/lib/kubelet/pods/b1f619c5-b367-4e79-959f-1e022fc48c23/volumes"
Oct 30 19:06:43.999518 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.999486    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"
Oct 30 19:06:43.999518 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.999508    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v" event={"ID":"9f961678-957e-4e77-94c2-c551cb6627cb","Type":"ContainerDied","Data":"a997e404756235f4054ba84110651822e0cdf5900c450f4104694600bb5308fa"}
Oct 30 19:06:43.999773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:43.999565    6014 scope.go:122] "RemoveContainer" containerID="9c59324f3efe53422e15b8633234278cb257be52251de28b0f57013d027564d1"
Oct 30 19:06:44.002860 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:44.002837    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"
Oct 30 19:06:44.002955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:44.002880    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq" event={"ID":"43fd9a91-35f5-454e-95ad-8f2c446ed639","Type":"ContainerDied","Data":"1c6fb4d7f6e72d8d508dc43c6436a1d27f3dff2d6ffcd1403605f6c74222121e"}
Oct 30 19:06:44.003031 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:44.003018    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"
Oct 30 19:06:44.011722 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:44.011699    6014 scope.go:122] "RemoveContainer" containerID="9c764953ac23ce7d8eda0fb182c49f72043b838338cf62b3f29526310a7fc221"
Oct 30 19:06:44.166349 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:44.166313    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"]
Oct 30 19:06:44.170273 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:44.170252    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-5549v"]
Oct 30 19:06:44.280647 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:44.280561    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"]
Oct 30 19:06:44.285663 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:44.285637    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-htc59"]
Oct 30 19:06:44.402384 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:44.402350    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"]
Oct 30 19:06:44.405399 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:44.405377    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-9p2tq"]
Oct 30 19:06:45.478443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:45.478413    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" path="/var/lib/kubelet/pods/0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c/volumes"
Oct 30 19:06:45.478984 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:45.478971    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43fd9a91-35f5-454e-95ad-8f2c446ed639" path="/var/lib/kubelet/pods/43fd9a91-35f5-454e-95ad-8f2c446ed639/volumes"
Oct 30 19:06:45.479452 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:45.479429    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f961678-957e-4e77-94c2-c551cb6627cb" path="/var/lib/kubelet/pods/9f961678-957e-4e77-94c2-c551cb6627cb/volumes"
Oct 30 19:06:51.951317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:06:51.951283    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"
Oct 30 19:06:53.259228 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:06:53.259193    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:07:02.049211 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:02.049174    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"]
Oct 30 19:07:02.098637 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:02.098598    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60b3957d-n9wtr"]
Oct 30 19:07:03.478665 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:03.478631    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0918e5bb-4103-4713-9f14-0848c3e37ab3" path="/var/lib/kubelet/pods/0918e5bb-4103-4713-9f14-0848c3e37ab3/volumes"
Oct 30 19:07:03.479205 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:03.479102    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:07:13.070443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:13.070408    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"]
Oct 30 19:07:13.093086 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:13.093056    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55a831a-8fnnx"]
Oct 30 19:07:13.478805 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:13.478731    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5962d0b-f82e-42d7-ae83-a482e5f32ef1" path="/var/lib/kubelet/pods/f5962d0b-f82e-42d7-ae83-a482e5f32ef1/volumes"
Oct 30 19:07:19.274979 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.274941    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"]
Oct 30 19:07:19.275886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.275865    6014 cpu_manager.go:414] "containerMap: removing container" podUID="2b35af62-6e06-4f5e-8c63-4b8a479f6845" containerName="istio-proxy"
Oct 30 19:07:19.275886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.275885    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b35af62-6e06-4f5e-8c63-4b8a479f6845" containerName="istio-proxy"
Oct 30 19:07:19.276047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.275899    6014 cpu_manager.go:414] "containerMap: removing container" podUID="43fd9a91-35f5-454e-95ad-8f2c446ed639" containerName="istio-proxy"
Oct 30 19:07:19.276047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.275908    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43fd9a91-35f5-454e-95ad-8f2c446ed639" containerName="istio-proxy"
Oct 30 19:07:19.276047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.275929    6014 cpu_manager.go:414] "containerMap: removing container" podUID="0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" containerName="istio-proxy"
Oct 30 19:07:19.276047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.275934    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" containerName="istio-proxy"
Oct 30 19:07:19.276047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.275976    6014 cpu_manager.go:414] "containerMap: removing container" podUID="9f961678-957e-4e77-94c2-c551cb6627cb" containerName="istio-proxy"
Oct 30 19:07:19.276047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.275983    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f961678-957e-4e77-94c2-c551cb6627cb" containerName="istio-proxy"
Oct 30 19:07:19.276281 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.276160    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f961678-957e-4e77-94c2-c551cb6627cb" containerName="istio-proxy"
Oct 30 19:07:19.276281 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.276178    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b35af62-6e06-4f5e-8c63-4b8a479f6845" containerName="istio-proxy"
Oct 30 19:07:19.276281 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.276184    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="43fd9a91-35f5-454e-95ad-8f2c446ed639" containerName="istio-proxy"
Oct 30 19:07:19.276281 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.276190    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f19a2ab-cc59-40f3-8d0f-b6ad0f76f12c" containerName="istio-proxy"
Oct 30 19:07:19.280381 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.280362    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"
Oct 30 19:07:19.304077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.304048    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e7cd214-n46qz\" (UID: \"af035e36-bc85-491a-a20b-b1451412fc15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"
Oct 30 19:07:19.304270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.304082    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e7cd214-n46qz\" (UID: \"af035e36-bc85-491a-a20b-b1451412fc15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"
Oct 30 19:07:19.304270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.304182    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbnb6\" (UniqueName: \"kubernetes.io/projected/af035e36-bc85-491a-a20b-b1451412fc15-kube-api-access-dbnb6\") pod \"maas-api-revoke-sub-8e7cd214-n46qz\" (UID: \"af035e36-bc85-491a-a20b-b1451412fc15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"
Oct 30 19:07:19.337496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.337463    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"]
Oct 30 19:07:19.380635 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.380596    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:07:19.380818 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.380662    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:07:19.406850 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.406806    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e7cd214-n46qz\" (UID: \"af035e36-bc85-491a-a20b-b1451412fc15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"
Oct 30 19:07:19.407050 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.406878    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e7cd214-n46qz\" (UID: \"af035e36-bc85-491a-a20b-b1451412fc15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"
Oct 30 19:07:19.407050 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.406936    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbnb6\" (UniqueName: \"kubernetes.io/projected/af035e36-bc85-491a-a20b-b1451412fc15-kube-api-access-dbnb6\") pod \"maas-api-revoke-sub-8e7cd214-n46qz\" (UID: \"af035e36-bc85-491a-a20b-b1451412fc15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"
Oct 30 19:07:19.408602 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.408570    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e7cd214-n46qz\" (UID: \"af035e36-bc85-491a-a20b-b1451412fc15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"
Oct 30 19:07:19.411875 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.411845    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e7cd214-n46qz\" (UID: \"af035e36-bc85-491a-a20b-b1451412fc15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"
Oct 30 19:07:19.466268 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.466239    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbnb6\" (UniqueName: \"kubernetes.io/projected/af035e36-bc85-491a-a20b-b1451412fc15-kube-api-access-dbnb6\") pod \"maas-api-revoke-sub-8e7cd214-n46qz\" (UID: \"af035e36-bc85-491a-a20b-b1451412fc15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"
Oct 30 19:07:19.595074 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.595040    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"
Oct 30 19:07:19.977611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:19.977587    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"]
Oct 30 19:07:19.978754 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:07:19.978730    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf035e36_bc85_491a_a20b_b1451412fc15.slice/crio-b7713c7de6ef20df6761a74fa42000227e4e13b255b36c6905198a3ad3ba4296 WatchSource:0}: Error finding container b7713c7de6ef20df6761a74fa42000227e4e13b255b36c6905198a3ad3ba4296: Status 404 returned error can't find the container with id b7713c7de6ef20df6761a74fa42000227e4e13b255b36c6905198a3ad3ba4296
Oct 30 19:07:20.431436 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:20.431395    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz" event={"ID":"af035e36-bc85-491a-a20b-b1451412fc15","Type":"ContainerStarted","Data":"b7713c7de6ef20df6761a74fa42000227e4e13b255b36c6905198a3ad3ba4296"}
Oct 30 19:07:21.049445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:21.049411    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"]
Oct 30 19:07:21.098836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:21.098800    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95358bf7-rrxbc"]
Oct 30 19:07:21.441730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:21.441697    6014 generic.go:356] "Generic (PLEG): container finished" podID="af035e36-bc85-491a-a20b-b1451412fc15" containerID="70fd3e1b9d129de52b35dff9ec8a7ca0e665dc040daa68876b9103188491ddc2" exitCode=0
Oct 30 19:07:21.442192 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:21.441779    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz" event={"ID":"af035e36-bc85-491a-a20b-b1451412fc15","Type":"ContainerDied","Data":"70fd3e1b9d129de52b35dff9ec8a7ca0e665dc040daa68876b9103188491ddc2"}
Oct 30 19:07:21.478248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:21.478221    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0062b2f7-32de-4f15-9c6c-7e50fe61584d" path="/var/lib/kubelet/pods/0062b2f7-32de-4f15-9c6c-7e50fe61584d/volumes"
Oct 30 19:07:22.646689 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:22.646657    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"
Oct 30 19:07:22.762584 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:22.762495    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af035e36-bc85-491a-a20b-b1451412fc15-kube-api-access-dbnb6\" (UniqueName: \"kubernetes.io/projected/af035e36-bc85-491a-a20b-b1451412fc15-kube-api-access-dbnb6\") pod \"af035e36-bc85-491a-a20b-b1451412fc15\" (UID: \"af035e36-bc85-491a-a20b-b1451412fc15\") "
Oct 30 19:07:22.762764 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:22.762622    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-openshift-ca\") pod \"af035e36-bc85-491a-a20b-b1451412fc15\" (UID: \"af035e36-bc85-491a-a20b-b1451412fc15\") "
Oct 30 19:07:22.762764 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:22.762673    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-xks-ca\") pod \"af035e36-bc85-491a-a20b-b1451412fc15\" (UID: \"af035e36-bc85-491a-a20b-b1451412fc15\") "
Oct 30 19:07:22.762982 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:22.762954    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-openshift-ca" pod "af035e36-bc85-491a-a20b-b1451412fc15" (UID: "af035e36-bc85-491a-a20b-b1451412fc15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:22.763495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:22.763473    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:07:22.764977 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:22.764950    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-xks-ca" pod "af035e36-bc85-491a-a20b-b1451412fc15" (UID: "af035e36-bc85-491a-a20b-b1451412fc15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:22.765062 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:22.764970    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af035e36-bc85-491a-a20b-b1451412fc15-kube-api-access-dbnb6" pod "af035e36-bc85-491a-a20b-b1451412fc15" (UID: "af035e36-bc85-491a-a20b-b1451412fc15"). InnerVolumeSpecName "kube-api-access-dbnb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:22.865189 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:22.865111    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbnb6\" (UniqueName: \"kubernetes.io/projected/af035e36-bc85-491a-a20b-b1451412fc15-kube-api-access-dbnb6\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:07:22.865435 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:22.865414    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af035e36-bc85-491a-a20b-b1451412fc15-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:07:23.339655 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:07:23.339617    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:07:23.463835 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:23.463798    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"
Oct 30 19:07:23.463983 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:23.463831    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz" event={"ID":"af035e36-bc85-491a-a20b-b1451412fc15","Type":"ContainerDied","Data":"b7713c7de6ef20df6761a74fa42000227e4e13b255b36c6905198a3ad3ba4296"}
Oct 30 19:07:23.463983 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:23.463904    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7713c7de6ef20df6761a74fa42000227e4e13b255b36c6905198a3ad3ba4296"
Oct 30 19:07:23.645035 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:23.644951    6014 scope.go:122] "RemoveContainer" containerID="2933bed322a852a5c33d91f6987e78f5ce353c027df8675c6295a98b2cb49956"
Oct 30 19:07:23.656883 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:23.656859    6014 scope.go:122] "RemoveContainer" containerID="5c5771ff6d1193c56cd5ccf857a767bd278a42d349ef9884c84dafb35c62b541"
Oct 30 19:07:23.667118 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:23.667099    6014 scope.go:122] "RemoveContainer" containerID="c7425960d3034298df279b045ccfbefbfb5d9b230db90d49e129235c8cc3633c"
Oct 30 19:07:23.677316 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:23.677293    6014 scope.go:122] "RemoveContainer" containerID="1144f5c9b248cadb44eb5d7bb884353b275b4a74c46e114dba0f849c8a509766"
Oct 30 19:07:23.687991 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:23.687973    6014 scope.go:122] "RemoveContainer" containerID="9a4fd2856512fc4a16f5bf53d5e4b0fcf21821571b35dc22014ee7c34f1423a1"
Oct 30 19:07:23.700856 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:23.700821    6014 scope.go:122] "RemoveContainer" containerID="74d3431fe88bc5a73e12ba4513308aa4c2ee37f2ce3e51549cd40bbc4a5a5ef9"
Oct 30 19:07:27.672550 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.672516    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"]
Oct 30 19:07:27.673445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.673426    6014 cpu_manager.go:414] "containerMap: removing container" podUID="af035e36-bc85-491a-a20b-b1451412fc15" containerName="revoke-keys"
Oct 30 19:07:27.673514 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.673448    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af035e36-bc85-491a-a20b-b1451412fc15" containerName="revoke-keys"
Oct 30 19:07:27.673704 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.673680    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="af035e36-bc85-491a-a20b-b1451412fc15" containerName="revoke-keys"
Oct 30 19:07:27.679749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.679723    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"
Oct 30 19:07:27.731698 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.731665    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"]
Oct 30 19:07:27.753658 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.753615    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p9tx\" (UniqueName: \"kubernetes.io/projected/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-kube-api-access-7p9tx\") pod \"maas-api-revoke-keys-e2e-ait-eb4c249f-26d68\" (UID: \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"
Oct 30 19:07:27.753909 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.753780    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb4c249f-26d68\" (UID: \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"
Oct 30 19:07:27.753998 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.753908    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb4c249f-26d68\" (UID: \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"
Oct 30 19:07:27.776029 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.775996    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:07:27.776238 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.776217    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:07:27.856020 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.855980    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb4c249f-26d68\" (UID: \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"
Oct 30 19:07:27.856239 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.856089    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb4c249f-26d68\" (UID: \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"
Oct 30 19:07:27.856301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.856284    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p9tx\" (UniqueName: \"kubernetes.io/projected/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-kube-api-access-7p9tx\") pod \"maas-api-revoke-keys-e2e-ait-eb4c249f-26d68\" (UID: \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"
Oct 30 19:07:27.856760 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.856731    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb4c249f-26d68\" (UID: \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"
Oct 30 19:07:27.858602 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.858582    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb4c249f-26d68\" (UID: \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"
Oct 30 19:07:27.913236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.913199    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p9tx\" (UniqueName: \"kubernetes.io/projected/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-kube-api-access-7p9tx\") pod \"maas-api-revoke-keys-e2e-ait-eb4c249f-26d68\" (UID: \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"
Oct 30 19:07:27.995128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:27.995045    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"
Oct 30 19:07:28.380604 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:28.380581    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"]
Oct 30 19:07:28.381587 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:07:28.381562    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb503c4d_46f8_44bb_85e0_1b21f3837c0a.slice/crio-7eb247ad086a875a46e9fe13769db59f960a067fa001e96cfe9bada061fc7611 WatchSource:0}: Error finding container 7eb247ad086a875a46e9fe13769db59f960a067fa001e96cfe9bada061fc7611: Status 404 returned error can't find the container with id 7eb247ad086a875a46e9fe13769db59f960a067fa001e96cfe9bada061fc7611
Oct 30 19:07:28.506783 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:28.506745    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68" event={"ID":"eb503c4d-46f8-44bb-85e0-1b21f3837c0a","Type":"ContainerStarted","Data":"7eb247ad086a875a46e9fe13769db59f960a067fa001e96cfe9bada061fc7611"}
Oct 30 19:07:29.517629 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:29.517592    6014 generic.go:356] "Generic (PLEG): container finished" podID="eb503c4d-46f8-44bb-85e0-1b21f3837c0a" containerID="90f17c40a12c2696c43c6b9301efe888320cc33c946b98eb5aa5b901be9e0524" exitCode=0
Oct 30 19:07:29.518097 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:29.517637    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68" event={"ID":"eb503c4d-46f8-44bb-85e0-1b21f3837c0a","Type":"ContainerDied","Data":"90f17c40a12c2696c43c6b9301efe888320cc33c946b98eb5aa5b901be9e0524"}
Oct 30 19:07:30.700608 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:30.700581    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"
Oct 30 19:07:30.812190 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:30.812065    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-kube-api-access-7p9tx\" (UniqueName: \"kubernetes.io/projected/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-kube-api-access-7p9tx\") pod \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\" (UID: \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\") "
Oct 30 19:07:30.812190 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:30.812123    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-xks-ca\") pod \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\" (UID: \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\") "
Oct 30 19:07:30.812438 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:30.812206    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-openshift-ca\") pod \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\" (UID: \"eb503c4d-46f8-44bb-85e0-1b21f3837c0a\") "
Oct 30 19:07:30.812586 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:30.812557    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-openshift-ca" pod "eb503c4d-46f8-44bb-85e0-1b21f3837c0a" (UID: "eb503c4d-46f8-44bb-85e0-1b21f3837c0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:30.812923 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:30.812904    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:07:30.814889 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:30.814859    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-xks-ca" pod "eb503c4d-46f8-44bb-85e0-1b21f3837c0a" (UID: "eb503c4d-46f8-44bb-85e0-1b21f3837c0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:30.815021 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:30.814974    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-kube-api-access-7p9tx" pod "eb503c4d-46f8-44bb-85e0-1b21f3837c0a" (UID: "eb503c4d-46f8-44bb-85e0-1b21f3837c0a"). InnerVolumeSpecName "kube-api-access-7p9tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:30.914723 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:30.914682    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p9tx\" (UniqueName: \"kubernetes.io/projected/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-kube-api-access-7p9tx\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:07:30.914723 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:30.914719    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb503c4d-46f8-44bb-85e0-1b21f3837c0a-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:07:31.538465 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:31.538436    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"
Oct 30 19:07:31.538465 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:31.538438    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68" event={"ID":"eb503c4d-46f8-44bb-85e0-1b21f3837c0a","Type":"ContainerDied","Data":"7eb247ad086a875a46e9fe13769db59f960a067fa001e96cfe9bada061fc7611"}
Oct 30 19:07:31.538711 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:31.538502    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7eb247ad086a875a46e9fe13769db59f960a067fa001e96cfe9bada061fc7611"
Oct 30 19:07:33.050794 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.050759    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"]
Oct 30 19:07:33.099178 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.099123    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed3861a-54974"]
Oct 30 19:07:33.478696 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.478665    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de838a97-cbed-4eca-ac85-26f9ebba6d9a" path="/var/lib/kubelet/pods/de838a97-cbed-4eca-ac85-26f9ebba6d9a/volumes"
Oct 30 19:07:33.842309 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.842275    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"]
Oct 30 19:07:33.843024 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.843010    6014 cpu_manager.go:414] "containerMap: removing container" podUID="eb503c4d-46f8-44bb-85e0-1b21f3837c0a" containerName="revoke-keys"
Oct 30 19:07:33.843069 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.843026    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb503c4d-46f8-44bb-85e0-1b21f3837c0a" containerName="revoke-keys"
Oct 30 19:07:33.843198 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.843188    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb503c4d-46f8-44bb-85e0-1b21f3837c0a" containerName="revoke-keys"
Oct 30 19:07:33.848876 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.848855    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:33.865155 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.865114    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-metrics-certs\") pod \"maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:33.865299 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.865271    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c954dfe7-18f9-40a9-a921-7c40f023ec98-openshift-service-ca\") pod \"maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:33.865574 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.865551    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zspqr\" (UniqueName: \"kubernetes.io/projected/c954dfe7-18f9-40a9-a921-7c40f023ec98-kube-api-access-zspqr\") pod \"maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:33.865629 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.865583    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-maas-api-tls\") pod \"maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:33.904321 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.904284    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4501c8d4-serving-cert\""
Oct 30 19:07:33.960033 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.959900    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"]
Oct 30 19:07:33.966961 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.966930    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c954dfe7-18f9-40a9-a921-7c40f023ec98-openshift-service-ca\") pod \"maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:33.967181 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.967128    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zspqr\" (UniqueName: \"kubernetes.io/projected/c954dfe7-18f9-40a9-a921-7c40f023ec98-kube-api-access-zspqr\") pod \"maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:33.967181 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.967173    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-maas-api-tls\") pod \"maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:33.967283 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.967264    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-metrics-certs\") pod \"maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:33.967334 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:07:33.967320    6014 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-serving-cert: secret "maas-api-e2e-mt-4501c8d4-serving-cert" not found
Oct 30 19:07:33.967398 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:07:33.967385    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-maas-api-tls podName:c954dfe7-18f9-40a9-a921-7c40f023ec98 nodeName:}" failed. No retries permitted until 2026-10-30 19:07:34.467370447 +0000 UTC m=+1693.110538039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-maas-api-tls") pod "maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg" (UID: "c954dfe7-18f9-40a9-a921-7c40f023ec98") : secret "maas-api-e2e-mt-4501c8d4-serving-cert" not found
Oct 30 19:07:33.967696 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.967672    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c954dfe7-18f9-40a9-a921-7c40f023ec98-openshift-service-ca\") pod \"maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:33.969867 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:33.969840    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-metrics-certs\") pod \"maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:34.024864 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:34.024831    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zspqr\" (UniqueName: \"kubernetes.io/projected/c954dfe7-18f9-40a9-a921-7c40f023ec98-kube-api-access-zspqr\") pod \"maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:34.475551 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:34.475514    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-maas-api-tls\") pod \"maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:34.478618 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:34.478586    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-maas-api-tls\") pod \"maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:34.763425 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:34.763338    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:35.147234 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:35.147203    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"]
Oct 30 19:07:35.148379 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:07:35.148344    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc954dfe7_18f9_40a9_a921_7c40f023ec98.slice/crio-d18dd57760fa69317e0f05ca3d45bcc50277e404d7351d3dfe0f5ce6e1f10fdb WatchSource:0}: Error finding container d18dd57760fa69317e0f05ca3d45bcc50277e404d7351d3dfe0f5ce6e1f10fdb: Status 404 returned error can't find the container with id d18dd57760fa69317e0f05ca3d45bcc50277e404d7351d3dfe0f5ce6e1f10fdb
Oct 30 19:07:35.572762 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:35.572726    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg" event={"ID":"c954dfe7-18f9-40a9-a921-7c40f023ec98","Type":"ContainerStarted","Data":"d18dd57760fa69317e0f05ca3d45bcc50277e404d7351d3dfe0f5ce6e1f10fdb"}
Oct 30 19:07:36.583824 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:36.583788    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg" event={"ID":"c954dfe7-18f9-40a9-a921-7c40f023ec98","Type":"ContainerStarted","Data":"e790c22f0c9799be4f908dad261786a6244ff89aa5db80f2759adaf8734c54b8"}
Oct 30 19:07:36.584304 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:36.583992    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:36.718879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:36.718823    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg" podStartSLOduration=2.811945273 podStartE2EDuration="3.718808931s" totalImagesPullingTime="906.863658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:36.715743893 +0000 UTC m=+1695.358911506" watchObservedRunningTime="2026-10-30 19:07:36.718808931 +0000 UTC m=+1695.361976544"
Oct 30 19:07:38.738548 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.738513    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"]
Oct 30 19:07:38.743332 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.743310    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:38.795229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.795188    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ae6a138d-serving-cert\""
Oct 30 19:07:38.851338 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.851301    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"]
Oct 30 19:07:38.851974 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.851947    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-maas-api-tls\") pod \"maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:38.852026 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.851990    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-metrics-certs\") pod \"maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:38.852211 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.852193    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5ghv\" (UniqueName: \"kubernetes.io/projected/d55d32d8-2bc9-4a98-98c1-a8701348f81b-kube-api-access-j5ghv\") pod \"maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:38.852436 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.852415    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55d32d8-2bc9-4a98-98c1-a8701348f81b-openshift-service-ca\") pod \"maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:38.954472 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.954431    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-maas-api-tls\") pod \"maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:38.954472 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.954471    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-metrics-certs\") pod \"maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:38.954765 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.954563    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5ghv\" (UniqueName: \"kubernetes.io/projected/d55d32d8-2bc9-4a98-98c1-a8701348f81b-kube-api-access-j5ghv\") pod \"maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:38.954765 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:07:38.954602    6014 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-serving-cert: secret "maas-api-e2e-mt-ae6a138d-serving-cert" not found
Oct 30 19:07:38.954765 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:07:38.954687    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-maas-api-tls podName:d55d32d8-2bc9-4a98-98c1-a8701348f81b nodeName:}" failed. No retries permitted until 2026-10-30 19:07:39.4546663 +0000 UTC m=+1698.097833895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-maas-api-tls") pod "maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj" (UID: "d55d32d8-2bc9-4a98-98c1-a8701348f81b") : secret "maas-api-e2e-mt-ae6a138d-serving-cert" not found
Oct 30 19:07:38.954924 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.954825    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55d32d8-2bc9-4a98-98c1-a8701348f81b-openshift-service-ca\") pod \"maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:38.955421 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.955395    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55d32d8-2bc9-4a98-98c1-a8701348f81b-openshift-service-ca\") pod \"maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:38.957225 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:38.957209    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-metrics-certs\") pod \"maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:39.012711 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:39.012633    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5ghv\" (UniqueName: \"kubernetes.io/projected/d55d32d8-2bc9-4a98-98c1-a8701348f81b-kube-api-access-j5ghv\") pod \"maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:39.466240 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:39.466198    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-maas-api-tls\") pod \"maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:39.468937 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:39.468896    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-maas-api-tls\") pod \"maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:39.657808 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:39.657776    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:40.045283 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:07:40.045249    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55d32d8_2bc9_4a98_98c1_a8701348f81b.slice/crio-1070593a5736ce515a4c49895b379780f52b6ee441b05d2bbbe8a96d8b659fed WatchSource:0}: Error finding container 1070593a5736ce515a4c49895b379780f52b6ee441b05d2bbbe8a96d8b659fed: Status 404 returned error can't find the container with id 1070593a5736ce515a4c49895b379780f52b6ee441b05d2bbbe8a96d8b659fed
Oct 30 19:07:40.045600 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:40.045532    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"]
Oct 30 19:07:40.619385 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:40.619352    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj" event={"ID":"d55d32d8-2bc9-4a98-98c1-a8701348f81b","Type":"ContainerStarted","Data":"1070593a5736ce515a4c49895b379780f52b6ee441b05d2bbbe8a96d8b659fed"}
Oct 30 19:07:41.632033 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:41.631995    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj" event={"ID":"d55d32d8-2bc9-4a98-98c1-a8701348f81b","Type":"ContainerStarted","Data":"7b16633ae8c24f809f8805996192940aa3b68d425fbbcfbea03493aa76bb3795"}
Oct 30 19:07:41.632461 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:41.632204    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:41.749051 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:41.748994    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj" podStartSLOduration=2.927493765 podStartE2EDuration="3.748978324s" totalImagesPullingTime="821.484559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:41.745119593 +0000 UTC m=+1700.388287202" watchObservedRunningTime="2026-10-30 19:07:41.748978324 +0000 UTC m=+1700.392145938"
Oct 30 19:07:42.598555 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:42.598528    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:07:43.724215 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:43.724177    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"]
Oct 30 19:07:43.730211 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:43.730093    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:43.805069 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:43.805030    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"]
Oct 30 19:07:43.842167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:43.838401    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-87ed12-serving-cert\""
Oct 30 19:07:43.893449 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:43.893407    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:43.893676 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:43.893566    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/455c906c-c097-4889-ac72-4c1dbc6b82d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:43.893676 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:43.893609    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq995\" (UniqueName: \"kubernetes.io/projected/455c906c-c097-4889-ac72-4c1dbc6b82d8-kube-api-access-zq995\") pod \"maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:43.893817 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:43.893752    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:43.998493 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:43.995923    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq995\" (UniqueName: \"kubernetes.io/projected/455c906c-c097-4889-ac72-4c1dbc6b82d8-kube-api-access-zq995\") pod \"maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:43.998493 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:43.996205    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:43.998493 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:43.996336    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:43.998493 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:43.996612    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/455c906c-c097-4889-ac72-4c1dbc6b82d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:43.998493 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:43.997663    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/455c906c-c097-4889-ac72-4c1dbc6b82d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:43.998493 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:07:43.998108    6014 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-serving-cert: secret "maas-api-e2e-shared-b-w1-87ed12-serving-cert" not found
Oct 30 19:07:43.999785 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:07:43.999080    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-maas-api-tls podName:455c906c-c097-4889-ac72-4c1dbc6b82d8 nodeName:}" failed. No retries permitted until 2026-10-30 19:07:44.499053566 +0000 UTC m=+1703.142221184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-maas-api-tls") pod "maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg" (UID: "455c906c-c097-4889-ac72-4c1dbc6b82d8") : secret "maas-api-e2e-shared-b-w1-87ed12-serving-cert" not found
Oct 30 19:07:44.008560 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:44.006799    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:44.060094 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:44.060060    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq995\" (UniqueName: \"kubernetes.io/projected/455c906c-c097-4889-ac72-4c1dbc6b82d8-kube-api-access-zq995\") pod \"maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:44.119071 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:44.119034    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:07:44.505105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:44.505067    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:44.507791 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:44.507762    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:44.673502 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:44.673455    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:45.056134 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:45.056102    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"]
Oct 30 19:07:45.057687 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:07:45.057658    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455c906c_c097_4889_ac72_4c1dbc6b82d8.slice/crio-94de94a195b259a5dd9848edb84b6ae4ef42d85d6eab2bd38e85194f2234b99e WatchSource:0}: Error finding container 94de94a195b259a5dd9848edb84b6ae4ef42d85d6eab2bd38e85194f2234b99e: Status 404 returned error can't find the container with id 94de94a195b259a5dd9848edb84b6ae4ef42d85d6eab2bd38e85194f2234b99e
Oct 30 19:07:45.681687 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:45.681643    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg" event={"ID":"455c906c-c097-4889-ac72-4c1dbc6b82d8","Type":"ContainerStarted","Data":"94de94a195b259a5dd9848edb84b6ae4ef42d85d6eab2bd38e85194f2234b99e"}
Oct 30 19:07:46.062795 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:46.062754    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"]
Oct 30 19:07:46.722077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:46.716330    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg" event={"ID":"455c906c-c097-4889-ac72-4c1dbc6b82d8","Type":"ContainerStarted","Data":"b83d41743f3495f4fae6f20c58bfd929194249fd41fbbc93537ab039cf82e9b3"}
Oct 30 19:07:46.722077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:46.716991    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:46.835814 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:46.835761    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg" podStartSLOduration=3.006159342 podStartE2EDuration="3.835745949s" totalImagesPullingTime="829.586607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:46.834769114 +0000 UTC m=+1705.477936728" watchObservedRunningTime="2026-10-30 19:07:46.835745949 +0000 UTC m=+1705.478913562"
Oct 30 19:07:47.091118 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:47.089255    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"]
Oct 30 19:07:47.118207 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:47.117387    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e3cf824-vnssl"]
Oct 30 19:07:47.523114 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:47.522918    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4427ca03-8e6c-496f-8426-4c5f160307b2" path="/var/lib/kubelet/pods/4427ca03-8e6c-496f-8426-4c5f160307b2/volumes"
Oct 30 19:07:47.687162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:47.686993    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:07:51.063700 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.063640    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5" podUID="bf265f8d-4e38-44b7-aba7-e7e7716f6579" containerName="payload-processing" containerID="cri-o://2d5623f10a48f84655e2e20707365fcb8d58d1d7e155f7afb15a73e25f5c5243" gracePeriod=25
Oct 30 19:07:51.351985 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.351947    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"
Oct 30 19:07:51.495419 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.495364    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf265f8d-4e38-44b7-aba7-e7e7716f6579-kube-api-access-99qhw\" (UniqueName: \"kubernetes.io/projected/bf265f8d-4e38-44b7-aba7-e7e7716f6579-kube-api-access-99qhw\") pod \"bf265f8d-4e38-44b7-aba7-e7e7716f6579\" (UID: \"bf265f8d-4e38-44b7-aba7-e7e7716f6579\") "
Oct 30 19:07:51.495742 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.495708    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf265f8d-4e38-44b7-aba7-e7e7716f6579-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf265f8d-4e38-44b7-aba7-e7e7716f6579-plugins-config-volume\") pod \"bf265f8d-4e38-44b7-aba7-e7e7716f6579\" (UID: \"bf265f8d-4e38-44b7-aba7-e7e7716f6579\") "
Oct 30 19:07:51.496081 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.496056    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf265f8d-4e38-44b7-aba7-e7e7716f6579-plugins-config-volume" pod "bf265f8d-4e38-44b7-aba7-e7e7716f6579" (UID: "bf265f8d-4e38-44b7-aba7-e7e7716f6579"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:51.496401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.496384    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf265f8d-4e38-44b7-aba7-e7e7716f6579-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:07:51.498321 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.498295    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf265f8d-4e38-44b7-aba7-e7e7716f6579-kube-api-access-99qhw" pod "bf265f8d-4e38-44b7-aba7-e7e7716f6579" (UID: "bf265f8d-4e38-44b7-aba7-e7e7716f6579"). InnerVolumeSpecName "kube-api-access-99qhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:51.598902 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.598823    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99qhw\" (UniqueName: \"kubernetes.io/projected/bf265f8d-4e38-44b7-aba7-e7e7716f6579-kube-api-access-99qhw\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:07:51.801405 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.801368    6014 generic.go:356] "Generic (PLEG): container finished" podID="bf265f8d-4e38-44b7-aba7-e7e7716f6579" containerID="2d5623f10a48f84655e2e20707365fcb8d58d1d7e155f7afb15a73e25f5c5243" exitCode=0
Oct 30 19:07:51.801585 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.801417    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"
Oct 30 19:07:51.801585 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.801457    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5" event={"ID":"bf265f8d-4e38-44b7-aba7-e7e7716f6579","Type":"ContainerDied","Data":"2d5623f10a48f84655e2e20707365fcb8d58d1d7e155f7afb15a73e25f5c5243"}
Oct 30 19:07:51.801585 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.801522    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5" event={"ID":"bf265f8d-4e38-44b7-aba7-e7e7716f6579","Type":"ContainerDied","Data":"7e744e980fedb24c15a48dbb4cde317f5e8c7e4fe4a28c38cd472183e4433385"}
Oct 30 19:07:51.801585 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.801551    6014 scope.go:122] "RemoveContainer" containerID="2d5623f10a48f84655e2e20707365fcb8d58d1d7e155f7afb15a73e25f5c5243"
Oct 30 19:07:51.812836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.812815    6014 scope.go:122] "RemoveContainer" containerID="2d5623f10a48f84655e2e20707365fcb8d58d1d7e155f7afb15a73e25f5c5243"
Oct 30 19:07:51.813227 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:07:51.813207    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5623f10a48f84655e2e20707365fcb8d58d1d7e155f7afb15a73e25f5c5243\": container with ID starting with 2d5623f10a48f84655e2e20707365fcb8d58d1d7e155f7afb15a73e25f5c5243 not found: ID does not exist" containerID="2d5623f10a48f84655e2e20707365fcb8d58d1d7e155f7afb15a73e25f5c5243"
Oct 30 19:07:51.813297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.813235    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5623f10a48f84655e2e20707365fcb8d58d1d7e155f7afb15a73e25f5c5243"} err="failed to get container status \"2d5623f10a48f84655e2e20707365fcb8d58d1d7e155f7afb15a73e25f5c5243\": rpc error: code = NotFound desc = could not find container \"2d5623f10a48f84655e2e20707365fcb8d58d1d7e155f7afb15a73e25f5c5243\": container with ID starting with 2d5623f10a48f84655e2e20707365fcb8d58d1d7e155f7afb15a73e25f5c5243 not found: ID does not exist"
Oct 30 19:07:51.967921 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:51.967840    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"]
Oct 30 19:07:52.017192 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:52.017157    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb4c249f-d7dccfbb5-vl7q5"]
Oct 30 19:07:52.781055 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:52.781028    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:07:52.879562 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:52.879528    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"]
Oct 30 19:07:52.881399 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:52.881370    6014 cpu_manager.go:414] "containerMap: removing container" podUID="bf265f8d-4e38-44b7-aba7-e7e7716f6579" containerName="payload-processing"
Oct 30 19:07:52.881399 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:52.881398    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf265f8d-4e38-44b7-aba7-e7e7716f6579" containerName="payload-processing"
Oct 30 19:07:52.881641 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:52.881626    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf265f8d-4e38-44b7-aba7-e7e7716f6579" containerName="payload-processing"
Oct 30 19:07:52.889560 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:52.889534    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"
Oct 30 19:07:52.942236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:52.942112    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4ebeba\""
Oct 30 19:07:52.942236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:52.942161    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4ebeba-dockercfg-btvpm\""
Oct 30 19:07:52.942482 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:52.942377    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"]
Oct 30 19:07:53.102166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:53.089397    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg9d4\" (UniqueName: \"kubernetes.io/projected/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-kube-api-access-lg9d4\") pod \"payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc\" (UID: \"577ef8da-25fd-49ff-bbd4-dc8e568fc99b\") " pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"
Oct 30 19:07:53.102166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:53.089568    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc\" (UID: \"577ef8da-25fd-49ff-bbd4-dc8e568fc99b\") " pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"
Oct 30 19:07:53.191897 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:53.191855    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg9d4\" (UniqueName: \"kubernetes.io/projected/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-kube-api-access-lg9d4\") pod \"payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc\" (UID: \"577ef8da-25fd-49ff-bbd4-dc8e568fc99b\") " pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"
Oct 30 19:07:53.192090 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:53.192030    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc\" (UID: \"577ef8da-25fd-49ff-bbd4-dc8e568fc99b\") " pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"
Oct 30 19:07:53.192988 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:53.192961    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc\" (UID: \"577ef8da-25fd-49ff-bbd4-dc8e568fc99b\") " pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"
Oct 30 19:07:53.249022 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:53.248995    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg9d4\" (UniqueName: \"kubernetes.io/projected/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-kube-api-access-lg9d4\") pod \"payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc\" (UID: \"577ef8da-25fd-49ff-bbd4-dc8e568fc99b\") " pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"
Oct 30 19:07:53.261629 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:07:53.261594    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:07:53.479646 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:53.479560    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf265f8d-4e38-44b7-aba7-e7e7716f6579" path="/var/lib/kubelet/pods/bf265f8d-4e38-44b7-aba7-e7e7716f6579/volumes"
Oct 30 19:07:53.529867 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:53.529829    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"
Oct 30 19:07:53.911414 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:53.911380    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"]
Oct 30 19:07:53.912603 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:07:53.912572    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577ef8da_25fd_49ff_bbd4_dc8e568fc99b.slice/crio-32e79d5c56eabebdf73fe5c56a0af43c8d17ae5d4255b228ba6ed253a9860d24 WatchSource:0}: Error finding container 32e79d5c56eabebdf73fe5c56a0af43c8d17ae5d4255b228ba6ed253a9860d24: Status 404 returned error can't find the container with id 32e79d5c56eabebdf73fe5c56a0af43c8d17ae5d4255b228ba6ed253a9860d24
Oct 30 19:07:54.903291 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:54.903261    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc" event={"ID":"577ef8da-25fd-49ff-bbd4-dc8e568fc99b","Type":"ContainerStarted","Data":"80988b742ffb21fd116c9977cca65ead6019a5b86aef38ced58d6be572c7c33d"}
Oct 30 19:07:54.903497 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:54.903316    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc" event={"ID":"577ef8da-25fd-49ff-bbd4-dc8e568fc99b","Type":"ContainerStarted","Data":"32e79d5c56eabebdf73fe5c56a0af43c8d17ae5d4255b228ba6ed253a9860d24"}
Oct 30 19:07:54.903497 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:54.903426    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"
Oct 30 19:07:55.030211 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:07:55.030154    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc" podStartSLOduration=3.03011864 podStartE2EDuration="3.03011864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:55.020840549 +0000 UTC m=+1713.664008164" watchObservedRunningTime="2026-10-30 19:07:55.03011864 +0000 UTC m=+1713.673286295"
Oct 30 19:08:02.131659 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.131579    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"]
Oct 30 19:08:02.137572 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.137546    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.250021 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.249986    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.250021 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.250027    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.250337 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.250088    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.250337 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.250275    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7e57df-ec86-4e01-adde-734560a4348e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.250337 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.250299    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.250337 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.250337    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.250548 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.250520    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7e57df-ec86-4e01-adde-734560a4348e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.250613 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.250598    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.250734 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.250713    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zb7\" (UniqueName: \"kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-kube-api-access-l6zb7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.344997 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.344954    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"]
Oct 30 19:08:02.353102 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.353061    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.353311 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.353129    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.353311 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.353192    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.353430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.353326    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7e57df-ec86-4e01-adde-734560a4348e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.353430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.353354    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.353430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.353414    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.353593 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.353538    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7e57df-ec86-4e01-adde-734560a4348e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.353653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.353605    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.353712 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.353640    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.353767 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.353710    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.353767 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.353725    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zb7\" (UniqueName: \"kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-kube-api-access-l6zb7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.354567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.354542    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7e57df-ec86-4e01-adde-734560a4348e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.354678 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.354592    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.354788 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.354766    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.356348 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.356316    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7e57df-ec86-4e01-adde-734560a4348e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.356538 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.356516    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.412180 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.412090    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.415001 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.414976    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zb7\" (UniqueName: \"kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-kube-api-access-l6zb7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fl4t5\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.452488 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.452457    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:02.890404 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.890374    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"]
Oct 30 19:08:02.890797 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:08:02.890769    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7e57df_ec86_4e01_adde_734560a4348e.slice/crio-b0b357f0ac9782e1fbb8e47e43c444245e40d3810c0a0838c1c113b744c34731 WatchSource:0}: Error finding container b0b357f0ac9782e1fbb8e47e43c444245e40d3810c0a0838c1c113b744c34731: Status 404 returned error can't find the container with id b0b357f0ac9782e1fbb8e47e43c444245e40d3810c0a0838c1c113b744c34731
Oct 30 19:08:02.893813 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.893776    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:02.893924 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.893853    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:02.893924 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.893883    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:02.967206 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:02.967173    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" event={"ID":"1a7e57df-ec86-4e01-adde-734560a4348e","Type":"ContainerStarted","Data":"b0b357f0ac9782e1fbb8e47e43c444245e40d3810c0a0838c1c113b744c34731"}
Oct 30 19:08:03.985628 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:03.985592    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" event={"ID":"1a7e57df-ec86-4e01-adde-734560a4348e","Type":"ContainerStarted","Data":"e958355e4a77859035cf69eb1a4a5f52a443f998d69d998a0100aa13c5a02aef"}
Oct 30 19:08:04.221748 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:04.221692    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" podStartSLOduration=2.221676846 podStartE2EDuration="2.221676846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:04.169638127 +0000 UTC m=+1722.812805745" watchObservedRunningTime="2026-10-30 19:08:04.221676846 +0000 UTC m=+1722.864844460"
Oct 30 19:08:04.467164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:04.456540    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:04.472299 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:04.471272    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fl4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:08:04.472299 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:04.471340    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" podUID="1a7e57df-ec86-4e01-adde-734560a4348e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 19:08:05.463622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:05.453495    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fl4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:08:05.463622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:05.453557    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" podUID="1a7e57df-ec86-4e01-adde-734560a4348e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 19:08:05.914721 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:05.914690    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"
Oct 30 19:08:06.083887 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:06.083848    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"]
Oct 30 19:08:06.090958 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:06.090929    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6cc08ed-6dct9"]
Oct 30 19:08:06.461663 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:06.461626    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fl4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:08:06.461952 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:06.461921    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" podUID="1a7e57df-ec86-4e01-adde-734560a4348e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 19:08:07.468165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:07.467543    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fl4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:08:07.468165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:07.467620    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" podUID="1a7e57df-ec86-4e01-adde-734560a4348e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 19:08:07.482783 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:07.482752    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f86f50-d5a0-4288-bc8f-0c45ea7fb845" path="/var/lib/kubelet/pods/e1f86f50-d5a0-4288-bc8f-0c45ea7fb845/volumes"
Oct 30 19:08:08.459166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:08.458542    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fl4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:08:08.459166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:08.458620    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" podUID="1a7e57df-ec86-4e01-adde-734560a4348e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 19:08:09.453601 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:09.453565    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fl4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:08:09.454050 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:09.453625    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" podUID="1a7e57df-ec86-4e01-adde-734560a4348e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 19:08:10.453765 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:10.453721    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fl4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:08:10.454335 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:10.453792    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" podUID="1a7e57df-ec86-4e01-adde-734560a4348e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 19:08:12.058749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:12.058703    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:12.059805 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:12.059770    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:12.060953 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:12.060931    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:08:22.828382 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:22.828349    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"]
Oct 30 19:08:22.828850 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:22.828559    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm" podUID="5278d58c-cb56-4847-88d1-e366fad02e4a" containerName="istio-proxy" containerID="cri-o://c4a4a5b55625e737dd50848daccb238c5a1a36ceabdd35a3e3a5ad39b8d511d8" gracePeriod=30
Oct 30 19:08:22.877573 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:22.877524    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"]
Oct 30 19:08:22.877815 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:22.877786    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7" podUID="461846ef-bc00-4cea-93f6-5d8deb63e544" containerName="istio-proxy" containerID="cri-o://b4777602ec192006168868ab0eb5eb49b3379fa82607e36ed889b41118113f52" gracePeriod=30
Oct 30 19:08:22.883281 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:22.883251    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"]
Oct 30 19:08:22.883519 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:22.883494    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4" podUID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerName="istio-proxy" containerID="cri-o://7e1134d28476871e6715914d521086e1a3a55ac3f76e0eab912ece02d4a0ecd3" gracePeriod=30
Oct 30 19:08:22.926697 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:22.926610    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"]
Oct 30 19:08:22.926843 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:22.926812    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy" containerID="cri-o://43353d2ed2af088db455094553290b8b0de9e4bcc5523c521bbea0293e61b756" gracePeriod=30
Oct 30 19:08:23.260240 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:08:23.260115    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:08:23.769919 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:23.769884    6014 scope.go:122] "RemoveContainer" containerID="e854df9d41a3ae34e456e73f6c447663db1143e4981de9185897a71440cc4d2e"
Oct 30 19:08:23.780814 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:23.780794    6014 scope.go:122] "RemoveContainer" containerID="e9ca279e8df5f6de9e86085edb875c130c46db3a22f52ab87b1cc49325c1c342"
Oct 30 19:08:23.790782 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:23.790764    6014 scope.go:122] "RemoveContainer" containerID="44a012876b307fee793e9abaf4c10769f755cc59acbcf98f7c7732f8a625055e"
Oct 30 19:08:23.800211 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:23.800186    6014 scope.go:122] "RemoveContainer" containerID="c8cdb4b536b8d01826b0de53c2276c5c5290e7df6b077ebcefc2e28b061c1749"
Oct 30 19:08:23.809718 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:23.809696    6014 scope.go:122] "RemoveContainer" containerID="4f3fbcd8724067fa6d6dad9bfc0c40353425934903ebd3198e895aecd34751fc"
Oct 30 19:08:23.819431 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:23.819400    6014 scope.go:122] "RemoveContainer" containerID="7502cc71ab581d311f1d876ff8a9d2e8d2d19dc4333c5088eeb192985480b7db"
Oct 30 19:08:26.868719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:26.868680    6014 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:08:26.869255 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:26.868776    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7" podUID="461846ef-bc00-4cea-93f6-5d8deb63e544" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:08:28.233921 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.233885    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:08:28.245362 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.245326    6014 generic.go:356] "Generic (PLEG): container finished" podID="42816184-e4b9-4216-8298-60ec4aebe37f" containerID="43353d2ed2af088db455094553290b8b0de9e4bcc5523c521bbea0293e61b756" exitCode=0
Oct 30 19:08:28.245362 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.245359    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" event={"ID":"42816184-e4b9-4216-8298-60ec4aebe37f","Type":"ContainerDied","Data":"43353d2ed2af088db455094553290b8b0de9e4bcc5523c521bbea0293e61b756"}
Oct 30 19:08:28.248042 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.248013    6014 generic.go:356] "Generic (PLEG): container finished" podID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerID="7e1134d28476871e6715914d521086e1a3a55ac3f76e0eab912ece02d4a0ecd3" exitCode=0
Oct 30 19:08:28.248224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.248094    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4" event={"ID":"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84","Type":"ContainerDied","Data":"7e1134d28476871e6715914d521086e1a3a55ac3f76e0eab912ece02d4a0ecd3"}
Oct 30 19:08:28.248224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.248217    6014 scope.go:122] "RemoveContainer" containerID="e5266cc0ab19505eff5c3e1dc325c6fc1b59970fac3e54aa6c1aa2fb765e62df"
Oct 30 19:08:28.253567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.253197    6014 generic.go:356] "Generic (PLEG): container finished" podID="461846ef-bc00-4cea-93f6-5d8deb63e544" containerID="b4777602ec192006168868ab0eb5eb49b3379fa82607e36ed889b41118113f52" exitCode=0
Oct 30 19:08:28.253567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.253465    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7" event={"ID":"461846ef-bc00-4cea-93f6-5d8deb63e544","Type":"ContainerDied","Data":"b4777602ec192006168868ab0eb5eb49b3379fa82607e36ed889b41118113f52"}
Oct 30 19:08:28.255690 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.255487    6014 generic.go:356] "Generic (PLEG): container finished" podID="5278d58c-cb56-4847-88d1-e366fad02e4a" containerID="c4a4a5b55625e737dd50848daccb238c5a1a36ceabdd35a3e3a5ad39b8d511d8" exitCode=0
Oct 30 19:08:28.255690 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.255543    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"
Oct 30 19:08:28.255690 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.255559    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm" event={"ID":"5278d58c-cb56-4847-88d1-e366fad02e4a","Type":"ContainerDied","Data":"c4a4a5b55625e737dd50848daccb238c5a1a36ceabdd35a3e3a5ad39b8d511d8"}
Oct 30 19:08:28.255690 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.255597    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm" event={"ID":"5278d58c-cb56-4847-88d1-e366fad02e4a","Type":"ContainerDied","Data":"4eed332239540d1448478845d4916ddc2dd9f50a6c5261fc46e72065fbf2db13"}
Oct 30 19:08:28.264979 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.264956    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 19:08:28.284348 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.284093    6014 scope.go:122] "RemoveContainer" containerID="c4a4a5b55625e737dd50848daccb238c5a1a36ceabdd35a3e3a5ad39b8d511d8"
Oct 30 19:08:28.299186 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.299121    6014 scope.go:122] "RemoveContainer" containerID="c4a4a5b55625e737dd50848daccb238c5a1a36ceabdd35a3e3a5ad39b8d511d8"
Oct 30 19:08:28.299605 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:08:28.299538    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a4a5b55625e737dd50848daccb238c5a1a36ceabdd35a3e3a5ad39b8d511d8\": container with ID starting with c4a4a5b55625e737dd50848daccb238c5a1a36ceabdd35a3e3a5ad39b8d511d8 not found: ID does not exist" containerID="c4a4a5b55625e737dd50848daccb238c5a1a36ceabdd35a3e3a5ad39b8d511d8"
Oct 30 19:08:28.299605 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.299574    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a4a5b55625e737dd50848daccb238c5a1a36ceabdd35a3e3a5ad39b8d511d8"} err="failed to get container status \"c4a4a5b55625e737dd50848daccb238c5a1a36ceabdd35a3e3a5ad39b8d511d8\": rpc error: code = NotFound desc = could not find container \"c4a4a5b55625e737dd50848daccb238c5a1a36ceabdd35a3e3a5ad39b8d511d8\": container with ID starting with c4a4a5b55625e737dd50848daccb238c5a1a36ceabdd35a3e3a5ad39b8d511d8 not found: ID does not exist"
Oct 30 19:08:28.345859 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.345781    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 19:08:28.347837 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.347811    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:08:28.369464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.369429    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-podinfo\") pod \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") "
Oct 30 19:08:28.369679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.369494    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5278d58c-cb56-4847-88d1-e366fad02e4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5278d58c-cb56-4847-88d1-e366fad02e4a-istiod-ca-cert\") pod \"5278d58c-cb56-4847-88d1-e366fad02e4a\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") "
Oct 30 19:08:28.369892 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.369862    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5278d58c-cb56-4847-88d1-e366fad02e4a-istiod-ca-cert" pod "5278d58c-cb56-4847-88d1-e366fad02e4a" (UID: "5278d58c-cb56-4847-88d1-e366fad02e4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:28.370020 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.369937    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-certs\") pod \"5278d58c-cb56-4847-88d1-e366fad02e4a\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") "
Oct 30 19:08:28.370299 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.370082    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-kube-api-access-wl8fs\" (UniqueName: \"kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-kube-api-access-wl8fs\") pod \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.370509    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-data\") pod \"5278d58c-cb56-4847-88d1-e366fad02e4a\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.370624    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-token\" (UniqueName: \"kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-token\") pod \"5278d58c-cb56-4847-88d1-e366fad02e4a\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.370669    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-socket\") pod \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.370752    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istiod-ca-cert\") pod \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.370809    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-token\" (UniqueName: \"kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-token\") pod \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.370900    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-data\") pod \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.371440    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-data" pod "5278d58c-cb56-4847-88d1-e366fad02e4a" (UID: "5278d58c-cb56-4847-88d1-e366fad02e4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.371042    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-socket\") pod \"5278d58c-cb56-4847-88d1-e366fad02e4a\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.371589    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-credential-socket\") pod \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.371694    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-certs\") pod \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.371730    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-podinfo\") pod \"5278d58c-cb56-4847-88d1-e366fad02e4a\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.371800    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-envoy\") pod \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\" (UID: \"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.371872    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-envoy\") pod \"5278d58c-cb56-4847-88d1-e366fad02e4a\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.371903    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istiod-ca-cert" pod "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" (UID: "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.371974    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-kube-api-access-vqjdg\" (UniqueName: \"kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-kube-api-access-vqjdg\") pod \"5278d58c-cb56-4847-88d1-e366fad02e4a\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.372011    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-credential-socket\") pod \"5278d58c-cb56-4847-88d1-e366fad02e4a\" (UID: \"5278d58c-cb56-4847-88d1-e366fad02e4a\") "
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.372118    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-data" pod "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" (UID: "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.372412    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-socket" pod "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" (UID: "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.372657    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-credential-socket" pod "5278d58c-cb56-4847-88d1-e366fad02e4a" (UID: "5278d58c-cb56-4847-88d1-e366fad02e4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.373044    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.373069    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5278d58c-cb56-4847-88d1-e366fad02e4a-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.373084    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.373096    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.373648    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.373836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.373668    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.375502 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.374037    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-credential-socket" pod "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" (UID: "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.375670 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.375614    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-socket" pod "5278d58c-cb56-4847-88d1-e366fad02e4a" (UID: "5278d58c-cb56-4847-88d1-e366fad02e4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.375837 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.375815    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-certs" pod "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" (UID: "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.376530 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.376499    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-podinfo" pod "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" (UID: "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:08:28.377263 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.377227    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-token" pod "5278d58c-cb56-4847-88d1-e366fad02e4a" (UID: "5278d58c-cb56-4847-88d1-e366fad02e4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.380232 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.380193    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-certs" pod "5278d58c-cb56-4847-88d1-e366fad02e4a" (UID: "5278d58c-cb56-4847-88d1-e366fad02e4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.380354 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.380338    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-envoy" pod "5278d58c-cb56-4847-88d1-e366fad02e4a" (UID: "5278d58c-cb56-4847-88d1-e366fad02e4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.380429 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.380391    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-kube-api-access-vqjdg" pod "5278d58c-cb56-4847-88d1-e366fad02e4a" (UID: "5278d58c-cb56-4847-88d1-e366fad02e4a"). InnerVolumeSpecName "kube-api-access-vqjdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.380429 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.380419    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-token" pod "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" (UID: "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.380562 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.380481    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-envoy" pod "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" (UID: "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.389384 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.389352    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-podinfo" pod "5278d58c-cb56-4847-88d1-e366fad02e4a" (UID: "5278d58c-cb56-4847-88d1-e366fad02e4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:08:28.389829 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.389803    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-kube-api-access-wl8fs" pod "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" (UID: "7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84"). InnerVolumeSpecName "kube-api-access-wl8fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.475651 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.475532    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-socket\") pod \"42816184-e4b9-4216-8298-60ec4aebe37f\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") "
Oct 30 19:08:28.475651 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.475627    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-certs\") pod \"461846ef-bc00-4cea-93f6-5d8deb63e544\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") "
Oct 30 19:08:28.476087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.475684    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-socket\") pod \"461846ef-bc00-4cea-93f6-5d8deb63e544\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") "
Oct 30 19:08:28.476087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.475779    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/461846ef-bc00-4cea-93f6-5d8deb63e544-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461846ef-bc00-4cea-93f6-5d8deb63e544-istiod-ca-cert\") pod \"461846ef-bc00-4cea-93f6-5d8deb63e544\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") "
Oct 30 19:08:28.476087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.475829    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-istio-token\" (UniqueName: \"kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-istio-token\") pod \"42816184-e4b9-4216-8298-60ec4aebe37f\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") "
Oct 30 19:08:28.476087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.475868    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-kube-api-access-cnhch\" (UniqueName: \"kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-kube-api-access-cnhch\") pod \"461846ef-bc00-4cea-93f6-5d8deb63e544\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") "
Oct 30 19:08:28.476087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.475923    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42816184-e4b9-4216-8298-60ec4aebe37f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42816184-e4b9-4216-8298-60ec4aebe37f-istiod-ca-cert\") pod \"42816184-e4b9-4216-8298-60ec4aebe37f\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") "
Oct 30 19:08:28.476087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.475951    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-envoy\") pod \"42816184-e4b9-4216-8298-60ec4aebe37f\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") "
Oct 30 19:08:28.481233 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.480430    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-socket" pod "42816184-e4b9-4216-8298-60ec4aebe37f" (UID: "42816184-e4b9-4216-8298-60ec4aebe37f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.481233 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.480588    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-certs" pod "461846ef-bc00-4cea-93f6-5d8deb63e544" (UID: "461846ef-bc00-4cea-93f6-5d8deb63e544"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.481510 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.481306    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-socket" pod "461846ef-bc00-4cea-93f6-5d8deb63e544" (UID: "461846ef-bc00-4cea-93f6-5d8deb63e544"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.482306 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.481636    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42816184-e4b9-4216-8298-60ec4aebe37f-istiod-ca-cert" pod "42816184-e4b9-4216-8298-60ec4aebe37f" (UID: "42816184-e4b9-4216-8298-60ec4aebe37f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:28.482306 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.482265    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461846ef-bc00-4cea-93f6-5d8deb63e544-istiod-ca-cert" pod "461846ef-bc00-4cea-93f6-5d8deb63e544" (UID: "461846ef-bc00-4cea-93f6-5d8deb63e544"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:28.483117 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.482562    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-envoy\") pod \"461846ef-bc00-4cea-93f6-5d8deb63e544\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") "
Oct 30 19:08:28.483117 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.482670    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-data\") pod \"42816184-e4b9-4216-8298-60ec4aebe37f\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") "
Oct 30 19:08:28.483117 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.482741    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42816184-e4b9-4216-8298-60ec4aebe37f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42816184-e4b9-4216-8298-60ec4aebe37f-istio-podinfo\") pod \"42816184-e4b9-4216-8298-60ec4aebe37f\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") "
Oct 30 19:08:28.483117 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.482819    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-token\" (UniqueName: \"kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-token\") pod \"461846ef-bc00-4cea-93f6-5d8deb63e544\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") "
Oct 30 19:08:28.483117 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.482900    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-certs\") pod \"42816184-e4b9-4216-8298-60ec4aebe37f\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") "
Oct 30 19:08:28.483117 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.482954    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-podinfo\") pod \"461846ef-bc00-4cea-93f6-5d8deb63e544\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") "
Oct 30 19:08:28.484685 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.484185    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-certs" pod "42816184-e4b9-4216-8298-60ec4aebe37f" (UID: "42816184-e4b9-4216-8298-60ec4aebe37f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.485165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.484914    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-credential-socket\") pod \"461846ef-bc00-4cea-93f6-5d8deb63e544\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") "
Oct 30 19:08:28.485165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.484987    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-kube-api-access-fxdk7\" (UniqueName: \"kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-kube-api-access-fxdk7\") pod \"42816184-e4b9-4216-8298-60ec4aebe37f\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") "
Oct 30 19:08:28.485165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.485061    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-data\") pod \"461846ef-bc00-4cea-93f6-5d8deb63e544\" (UID: \"461846ef-bc00-4cea-93f6-5d8deb63e544\") "
Oct 30 19:08:28.486378 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.485389    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-credential-socket\") pod \"42816184-e4b9-4216-8298-60ec4aebe37f\" (UID: \"42816184-e4b9-4216-8298-60ec4aebe37f\") "
Oct 30 19:08:28.486378 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486156    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-credential-socket" pod "461846ef-bc00-4cea-93f6-5d8deb63e544" (UID: "461846ef-bc00-4cea-93f6-5d8deb63e544"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.486737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486573    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.486737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486600    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.486737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486615    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.486737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486630    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.486737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486643    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.486737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486656    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqjdg\" (UniqueName: \"kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-kube-api-access-vqjdg\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.486737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486669    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.486737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486684    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.486737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486700    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl8fs\" (UniqueName: \"kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-kube-api-access-wl8fs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.486737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486715    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.486737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486729    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.486737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486742    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5278d58c-cb56-4847-88d1-e366fad02e4a-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.487567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486756    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.487567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486771    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461846ef-bc00-4cea-93f6-5d8deb63e544-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.487567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486785    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.487567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486799    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42816184-e4b9-4216-8298-60ec4aebe37f-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.487567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486812    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5278d58c-cb56-4847-88d1-e366fad02e4a-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.487567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486825    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.487567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486841    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.487567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486576    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-credential-socket" pod "42816184-e4b9-4216-8298-60ec4aebe37f" (UID: "42816184-e4b9-4216-8298-60ec4aebe37f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.487567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.486971    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-istio-token" pod "42816184-e4b9-4216-8298-60ec4aebe37f" (UID: "42816184-e4b9-4216-8298-60ec4aebe37f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.488458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.488420    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-data" pod "461846ef-bc00-4cea-93f6-5d8deb63e544" (UID: "461846ef-bc00-4cea-93f6-5d8deb63e544"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.489263 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.489239    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-data" pod "42816184-e4b9-4216-8298-60ec4aebe37f" (UID: "42816184-e4b9-4216-8298-60ec4aebe37f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.490481 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.490434    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-token" pod "461846ef-bc00-4cea-93f6-5d8deb63e544" (UID: "461846ef-bc00-4cea-93f6-5d8deb63e544"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.491756 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.491721    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-kube-api-access-cnhch" pod "461846ef-bc00-4cea-93f6-5d8deb63e544" (UID: "461846ef-bc00-4cea-93f6-5d8deb63e544"). InnerVolumeSpecName "kube-api-access-cnhch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.492060 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.491880    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-envoy" pod "461846ef-bc00-4cea-93f6-5d8deb63e544" (UID: "461846ef-bc00-4cea-93f6-5d8deb63e544"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.492060 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.491952    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-envoy" pod "42816184-e4b9-4216-8298-60ec4aebe37f" (UID: "42816184-e4b9-4216-8298-60ec4aebe37f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.492060 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.492021    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42816184-e4b9-4216-8298-60ec4aebe37f-istio-podinfo" pod "42816184-e4b9-4216-8298-60ec4aebe37f" (UID: "42816184-e4b9-4216-8298-60ec4aebe37f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:08:28.492815 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.492780    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-kube-api-access-fxdk7" pod "42816184-e4b9-4216-8298-60ec4aebe37f" (UID: "42816184-e4b9-4216-8298-60ec4aebe37f"). InnerVolumeSpecName "kube-api-access-fxdk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.493300 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.493281    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-podinfo" pod "461846ef-bc00-4cea-93f6-5d8deb63e544" (UID: "461846ef-bc00-4cea-93f6-5d8deb63e544"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:08:28.588315 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.588283    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxdk7\" (UniqueName: \"kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-kube-api-access-fxdk7\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.588315 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.588316    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.588576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.588330    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.588576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.588344    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42816184-e4b9-4216-8298-60ec4aebe37f-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.588576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.588357    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnhch\" (UniqueName: \"kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-kube-api-access-cnhch\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.588576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.588370    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.588576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.588383    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.588576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.588395    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42816184-e4b9-4216-8298-60ec4aebe37f-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.588576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.588407    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42816184-e4b9-4216-8298-60ec4aebe37f-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.588576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.588420    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.588576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.588433    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461846ef-bc00-4cea-93f6-5d8deb63e544-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.901560 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.901523    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"]
Oct 30 19:08:28.905352 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:28.905322    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-h85nm"]
Oct 30 19:08:29.264788 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.264703    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"
Oct 30 19:08:29.264788 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.264736    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4" event={"ID":"7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84","Type":"ContainerDied","Data":"a40b3f279c8c5685a81fecd2d4d5e76abc3d1a1554a25d08636660e776f23e93"}
Oct 30 19:08:29.265311 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.264805    6014 scope.go:122] "RemoveContainer" containerID="7e1134d28476871e6715914d521086e1a3a55ac3f76e0eab912ece02d4a0ecd3"
Oct 30 19:08:29.266338 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.266320    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"
Oct 30 19:08:29.266482 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.266341    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7" event={"ID":"461846ef-bc00-4cea-93f6-5d8deb63e544","Type":"ContainerDied","Data":"820d5a87087acffbe12dd32c5617feb6c327886de56d59ee16e7a80cb112caa5"}
Oct 30 19:08:29.275187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.275121    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr" event={"ID":"42816184-e4b9-4216-8298-60ec4aebe37f","Type":"ContainerDied","Data":"c3f31d0031abf2d633765af024927e663662d850cd50ebb9aca74326f65076c6"}
Oct 30 19:08:29.275303 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.275196    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"
Oct 30 19:08:29.281340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.281319    6014 scope.go:122] "RemoveContainer" containerID="b4777602ec192006168868ab0eb5eb49b3379fa82607e36ed889b41118113f52"
Oct 30 19:08:29.291847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.291827    6014 scope.go:122] "RemoveContainer" containerID="43353d2ed2af088db455094553290b8b0de9e4bcc5523c521bbea0293e61b756"
Oct 30 19:08:29.478595 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.478566    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5278d58c-cb56-4847-88d1-e366fad02e4a" path="/var/lib/kubelet/pods/5278d58c-cb56-4847-88d1-e366fad02e4a/volumes"
Oct 30 19:08:29.495126 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.495085    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"]
Oct 30 19:08:29.499674 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.499653    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-s4km7"]
Oct 30 19:08:29.606770 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.606728    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"]
Oct 30 19:08:29.614202 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.614165    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-cjptr"]
Oct 30 19:08:29.720707 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.720655    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"]
Oct 30 19:08:29.724794 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.724764    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-hqqw4"]
Oct 30 19:08:29.941736 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:29.941647    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:08:31.480780 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:31.480751    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" path="/var/lib/kubelet/pods/42816184-e4b9-4216-8298-60ec4aebe37f/volumes"
Oct 30 19:08:31.481335 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:31.481319    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="461846ef-bc00-4cea-93f6-5d8deb63e544" path="/var/lib/kubelet/pods/461846ef-bc00-4cea-93f6-5d8deb63e544/volumes"
Oct 30 19:08:31.481814 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:31.481801    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" path="/var/lib/kubelet/pods/7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84/volumes"
Oct 30 19:08:36.279347 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:36.279304    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"]
Oct 30 19:08:36.328103 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:36.328071    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb4c249f-26d68"]
Oct 30 19:08:37.226440 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.226409    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"]
Oct 30 19:08:37.227295 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.227276    6014 cpu_manager.go:414] "containerMap: removing container" podUID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerName="istio-proxy"
Oct 30 19:08:37.227295 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.227295    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerName="istio-proxy"
Oct 30 19:08:37.227473 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.227304    6014 cpu_manager.go:414] "containerMap: removing container" podUID="461846ef-bc00-4cea-93f6-5d8deb63e544" containerName="istio-proxy"
Oct 30 19:08:37.227473 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.227308    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461846ef-bc00-4cea-93f6-5d8deb63e544" containerName="istio-proxy"
Oct 30 19:08:37.227473 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.227337    6014 cpu_manager.go:414] "containerMap: removing container" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy"
Oct 30 19:08:37.227473 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.227343    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy"
Oct 30 19:08:37.227473 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.227358    6014 cpu_manager.go:414] "containerMap: removing container" podUID="5278d58c-cb56-4847-88d1-e366fad02e4a" containerName="istio-proxy"
Oct 30 19:08:37.227473 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.227363    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5278d58c-cb56-4847-88d1-e366fad02e4a" containerName="istio-proxy"
Oct 30 19:08:37.227653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.227491    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="42816184-e4b9-4216-8298-60ec4aebe37f" containerName="istio-proxy"
Oct 30 19:08:37.227653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.227502    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="5278d58c-cb56-4847-88d1-e366fad02e4a" containerName="istio-proxy"
Oct 30 19:08:37.227653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.227514    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerName="istio-proxy"
Oct 30 19:08:37.227653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.227520    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="461846ef-bc00-4cea-93f6-5d8deb63e544" containerName="istio-proxy"
Oct 30 19:08:37.227653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.227524    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerName="istio-proxy"
Oct 30 19:08:37.233363 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.233326    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.285597 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.285567    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f26b1d-openshift-default-dockercfg-nz7f9\""
Oct 30 19:08:37.345902 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.345858    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"]
Oct 30 19:08:37.354916 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.354880    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7pcd\" (UniqueName: \"kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-kube-api-access-z7pcd\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.370224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.369463    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-credential-socket\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.370557 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.370505    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istiod-ca-cert\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.370864 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.370845    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-socket\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.371028 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.371015    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-certs\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.371209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.371192    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-token\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.371495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.371439    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-podinfo\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.371589 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.371531    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-data\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.371713 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.371587    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-envoy\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.473813 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.473775    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-certs\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.474009 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.473880    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-token\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.474009 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.473967    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-podinfo\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.474133 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.474008    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-data\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.474133 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.474076    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-envoy\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.474262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.474184    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-certs\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.474319 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.474255    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7pcd\" (UniqueName: \"kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-kube-api-access-z7pcd\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.474369 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.474344    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-credential-socket\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.474495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.474392    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istiod-ca-cert\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.474495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.474412    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-data\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.474578 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.474551    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-socket\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.474677 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.474655    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-credential-socket\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.475210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.475185    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-socket\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.475327 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.475271    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istiod-ca-cert\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.476725 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.476672    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-envoy\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.477076 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.477055    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-podinfo\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.478917 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.478895    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb503c4d-46f8-44bb-85e0-1b21f3837c0a" path="/var/lib/kubelet/pods/eb503c4d-46f8-44bb-85e0-1b21f3837c0a/volumes"
Oct 30 19:08:37.530727 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.530695    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-token\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.530913 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.530799    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7pcd\" (UniqueName: \"kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-kube-api-access-z7pcd\") pod \"e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") " pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:37.549055 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:37.549015    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:38.011975 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:38.011950    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"]
Oct 30 19:08:38.013539 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:08:38.013512    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d8cf2e_7a19_49f8_8967_2deeff9ed05c.slice/crio-cb2222da3d91a5e3e9f2fad6245cb4d13b87f79afadcf76f6b0eee0daa3608a3 WatchSource:0}: Error finding container cb2222da3d91a5e3e9f2fad6245cb4d13b87f79afadcf76f6b0eee0daa3608a3: Status 404 returned error can't find the container with id cb2222da3d91a5e3e9f2fad6245cb4d13b87f79afadcf76f6b0eee0daa3608a3
Oct 30 19:08:38.016783 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:38.016741    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:38.016889 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:38.016821    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:38.016889 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:38.016853    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:38.361099 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:38.361066    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" event={"ID":"69d8cf2e-7a19-49f8-8967-2deeff9ed05c","Type":"ContainerStarted","Data":"61437ab78b927ffa560c1d02fc53c5f1fd9f3b0e6a4e804208b27d7b89eb9c39"}
Oct 30 19:08:38.361625 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:38.361126    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" event={"ID":"69d8cf2e-7a19-49f8-8967-2deeff9ed05c","Type":"ContainerStarted","Data":"cb2222da3d91a5e3e9f2fad6245cb4d13b87f79afadcf76f6b0eee0daa3608a3"}
Oct 30 19:08:38.482102 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:38.482053    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" podStartSLOduration=1.482038143 podStartE2EDuration="1.482038143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:38.478951439 +0000 UTC m=+1757.122119053" watchObservedRunningTime="2026-10-30 19:08:38.482038143 +0000 UTC m=+1757.125205756"
Oct 30 19:08:38.549347 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:38.549316    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:39.556056 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:39.556024    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:40.383225 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:40.383186    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:40.384350 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:40.384330    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:08:43.298167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.297476    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"]
Oct 30 19:08:43.298955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.298711    6014 cpu_manager.go:414] "containerMap: removing container" podUID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerName="istio-proxy"
Oct 30 19:08:43.298955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.298735    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9eb37d-8fc9-4b7d-90f9-6792ecbd7e84" containerName="istio-proxy"
Oct 30 19:08:43.307163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.304851    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.383173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.379568    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0b8d0f-openshift-default-dockercfg-jcvsm\""
Oct 30 19:08:43.394184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.393893    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-data\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.394184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.393977    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-envoy\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.394184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.394057    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a2f9fa-4621-45b8-84a8-2aa9721df791-istiod-ca-cert\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.394184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.394098    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-socket\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.395236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.394645    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr2mn\" (UniqueName: \"kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-kube-api-access-pr2mn\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.395236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.394814    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-podinfo\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.395236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.394856    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-certs\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.395236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.394926    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-credential-socket\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.395236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.395085    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-token\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.445807 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.445766    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"]
Oct 30 19:08:43.496528 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.496497    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-credential-socket\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.496857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.496839    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-token\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.497003 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.496989    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-data\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.497127 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.497115    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-envoy\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.497314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.497297    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a2f9fa-4621-45b8-84a8-2aa9721df791-istiod-ca-cert\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.497438 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.497425    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-socket\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.497565 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.497554    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr2mn\" (UniqueName: \"kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-kube-api-access-pr2mn\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.497769 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.497757    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-podinfo\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.497872 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.497861    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-certs\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.498413 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.498392    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-certs\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.498765 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.498745    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-credential-socket\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.499410 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.499381    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-data\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.500448 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.500427    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-socket\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.501160 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.501123    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a2f9fa-4621-45b8-84a8-2aa9721df791-istiod-ca-cert\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.504009 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.503990    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-podinfo\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.504329 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.504299    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-envoy\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.557617 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.557537    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-token\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.558558 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.558535    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr2mn\" (UniqueName: \"kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-kube-api-access-pr2mn\") pod \"e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") " pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:43.641742 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:43.641705    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:44.119669 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:44.119620    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"]
Oct 30 19:08:44.124852 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:08:44.124822    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a2f9fa_4621_45b8_84a8_2aa9721df791.slice/crio-ad3d70e6d3bd3eb9907aa807ac69dce2d6acf7c6cb9be943fbd41e4dbaacb9fc WatchSource:0}: Error finding container ad3d70e6d3bd3eb9907aa807ac69dce2d6acf7c6cb9be943fbd41e4dbaacb9fc: Status 404 returned error can't find the container with id ad3d70e6d3bd3eb9907aa807ac69dce2d6acf7c6cb9be943fbd41e4dbaacb9fc
Oct 30 19:08:44.128751 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:44.128709    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:44.128859 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:44.128814    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:44.128909 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:44.128859    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:44.461990 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:44.461896    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" event={"ID":"07a2f9fa-4621-45b8-84a8-2aa9721df791","Type":"ContainerStarted","Data":"2e7bbe2992568a048be8347dfa6ed2188d3b0ee8b67ef2d45d6f28630e0ef7a2"}
Oct 30 19:08:44.461990 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:44.461967    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" event={"ID":"07a2f9fa-4621-45b8-84a8-2aa9721df791","Type":"ContainerStarted","Data":"ad3d70e6d3bd3eb9907aa807ac69dce2d6acf7c6cb9be943fbd41e4dbaacb9fc"}
Oct 30 19:08:44.586361 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:44.586287    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" podStartSLOduration=1.586266221 podStartE2EDuration="1.586266221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:44.580652041 +0000 UTC m=+1763.223819657" watchObservedRunningTime="2026-10-30 19:08:44.586266221 +0000 UTC m=+1763.229433835"
Oct 30 19:08:44.642328 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:44.642290    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:45.648850 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:45.648818    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:46.488999 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:46.488962    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:46.492412 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:46.492377    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:08:53.267765 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:08:53.267724    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:08:54.004464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:54.004424    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"]
Oct 30 19:08:54.004765 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:54.004646    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" podUID="898fcd26-1c1e-40e6-994d-619340f782cb" containerName="istio-proxy" containerID="cri-o://2f0282cc95f3af422d3a83d09c30f8f687312f19747accc2a3a8b495bfe36093" gracePeriod=30
Oct 30 19:08:54.038970 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:54.038928    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"]
Oct 30 19:08:54.039285 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:54.039257    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy" containerID="cri-o://4640b3770dffad6254fb51f6b4ac62d506f1d92c9e5f105bdc00d2145aa8e026" gracePeriod=30
Oct 30 19:08:54.053659 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:54.053626    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"]
Oct 30 19:08:54.053857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:54.053835    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy" containerID="cri-o://3a67e298f1ea43f5a7100542fcbfef257a2602621e1c1bea56569811822761a5" gracePeriod=30
Oct 30 19:08:58.508325 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.508289    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:08:58.508756 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.508356    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" podUID="898fcd26-1c1e-40e6-994d-619340f782cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:08:58.532513 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.532482    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"]
Oct 30 19:08:58.537514 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.537492    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:58.598787 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.598750    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"]
Oct 30 19:08:58.617973 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.617921    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4ebeba-serving-cert\""
Oct 30 19:08:58.684026 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.683993    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85e7d51-316a-4e68-87e0-03217deade17-openshift-service-ca\") pod \"maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:58.684258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.684237    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-maas-api-tls\") pod \"maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:58.684323 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.684304    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-metrics-certs\") pod \"maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:58.684378 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.684332    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dldxv\" (UniqueName: \"kubernetes.io/projected/b85e7d51-316a-4e68-87e0-03217deade17-kube-api-access-dldxv\") pod \"maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:58.785405 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.785320    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85e7d51-316a-4e68-87e0-03217deade17-openshift-service-ca\") pod \"maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:58.785588 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.785518    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-maas-api-tls\") pod \"maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:58.785588 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.785578    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-metrics-certs\") pod \"maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:58.785714 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.785604    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dldxv\" (UniqueName: \"kubernetes.io/projected/b85e7d51-316a-4e68-87e0-03217deade17-kube-api-access-dldxv\") pod \"maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:58.785714 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:08:58.785656    6014 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-serving-cert: secret "maas-api-e2e-api-a-4ebeba-serving-cert" not found
Oct 30 19:08:58.785825 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:08:58.785722    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-maas-api-tls podName:b85e7d51-316a-4e68-87e0-03217deade17 nodeName:}" failed. No retries permitted until 2026-10-30 19:08:59.285707686 +0000 UTC m=+1777.928875278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-maas-api-tls") pod "maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9" (UID: "b85e7d51-316a-4e68-87e0-03217deade17") : secret "maas-api-e2e-api-a-4ebeba-serving-cert" not found
Oct 30 19:08:58.786101 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.786076    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85e7d51-316a-4e68-87e0-03217deade17-openshift-service-ca\") pod \"maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:58.788306 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.788282    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-metrics-certs\") pod \"maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:58.901981 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:58.901941    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dldxv\" (UniqueName: \"kubernetes.io/projected/b85e7d51-316a-4e68-87e0-03217deade17-kube-api-access-dldxv\") pod \"maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:59.244643 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.244604    6014 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:08:59.244834 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.244682    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:08:59.301313 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.300802    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-maas-api-tls\") pod \"maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:59.308450 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.308338    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-maas-api-tls\") pod \"maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:59.387961 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.386285    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:08:59.464763 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.464460    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:08:59.501713 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.501686    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:08:59.505489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.504832    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:08:59.505631 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.505582    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-certs\") pod \"58b2dba2-c676-4658-8dc3-d47d9782b180\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") "
Oct 30 19:08:59.505782 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.505738    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-socket\") pod \"58b2dba2-c676-4658-8dc3-d47d9782b180\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") "
Oct 30 19:08:59.506223 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.505841    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-podinfo\") pod \"58b2dba2-c676-4658-8dc3-d47d9782b180\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") "
Oct 30 19:08:59.506223 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.505924    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-data\") pod \"58b2dba2-c676-4658-8dc3-d47d9782b180\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") "
Oct 30 19:08:59.506223 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.506101    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-token\" (UniqueName: \"kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-token\") pod \"58b2dba2-c676-4658-8dc3-d47d9782b180\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") "
Oct 30 19:08:59.506954 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.506469    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-envoy\") pod \"58b2dba2-c676-4658-8dc3-d47d9782b180\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") "
Oct 30 19:08:59.506954 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.506679    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-kube-api-access-q9dqh\" (UniqueName: \"kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-kube-api-access-q9dqh\") pod \"58b2dba2-c676-4658-8dc3-d47d9782b180\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") "
Oct 30 19:08:59.506954 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.506870    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58b2dba2-c676-4658-8dc3-d47d9782b180-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b2dba2-c676-4658-8dc3-d47d9782b180-istiod-ca-cert\") pod \"58b2dba2-c676-4658-8dc3-d47d9782b180\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") "
Oct 30 19:08:59.507353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.506988    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-credential-socket\") pod \"58b2dba2-c676-4658-8dc3-d47d9782b180\" (UID: \"58b2dba2-c676-4658-8dc3-d47d9782b180\") "
Oct 30 19:08:59.509623 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.508888    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-data" pod "58b2dba2-c676-4658-8dc3-d47d9782b180" (UID: "58b2dba2-c676-4658-8dc3-d47d9782b180"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.509623 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.509123    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-certs" pod "58b2dba2-c676-4658-8dc3-d47d9782b180" (UID: "58b2dba2-c676-4658-8dc3-d47d9782b180"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.509623 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.509343    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-socket" pod "58b2dba2-c676-4658-8dc3-d47d9782b180" (UID: "58b2dba2-c676-4658-8dc3-d47d9782b180"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.513758 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.513597    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-podinfo" pod "58b2dba2-c676-4658-8dc3-d47d9782b180" (UID: "58b2dba2-c676-4658-8dc3-d47d9782b180"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:08:59.514697 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.514649    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-credential-socket" pod "58b2dba2-c676-4658-8dc3-d47d9782b180" (UID: "58b2dba2-c676-4658-8dc3-d47d9782b180"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.515378 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.515273    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b2dba2-c676-4658-8dc3-d47d9782b180-istiod-ca-cert" pod "58b2dba2-c676-4658-8dc3-d47d9782b180" (UID: "58b2dba2-c676-4658-8dc3-d47d9782b180"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:59.529198 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.519259    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.529198 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.519342    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.529198 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.519364    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.529198 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.519386    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.529198 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.519416    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b2dba2-c676-4658-8dc3-d47d9782b180-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.532653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.532566    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-token" pod "58b2dba2-c676-4658-8dc3-d47d9782b180" (UID: "58b2dba2-c676-4658-8dc3-d47d9782b180"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:59.533127 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.533095    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-kube-api-access-q9dqh" pod "58b2dba2-c676-4658-8dc3-d47d9782b180" (UID: "58b2dba2-c676-4658-8dc3-d47d9782b180"). InnerVolumeSpecName "kube-api-access-q9dqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:59.533655 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.533629    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-envoy" pod "58b2dba2-c676-4658-8dc3-d47d9782b180" (UID: "58b2dba2-c676-4658-8dc3-d47d9782b180"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.622812 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.622648    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-data\") pod \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") "
Oct 30 19:08:59.626187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.623797    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-data" pod "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" (UID: "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.626187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.625524    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-socket\") pod \"898fcd26-1c1e-40e6-994d-619340f782cb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") "
Oct 30 19:08:59.626187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.625569    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-certs\") pod \"898fcd26-1c1e-40e6-994d-619340f782cb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") "
Oct 30 19:08:59.626187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.625653    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-podinfo\") pod \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") "
Oct 30 19:08:59.626187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.625683    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istiod-ca-cert\") pod \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") "
Oct 30 19:08:59.626187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.625735    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-socket\") pod \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") "
Oct 30 19:08:59.626187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.625780    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-token\" (UniqueName: \"kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-token\") pod \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") "
Oct 30 19:08:59.626187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.625809    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-kube-api-access-msphp\" (UniqueName: \"kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-kube-api-access-msphp\") pod \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") "
Oct 30 19:08:59.626187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.625912    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/898fcd26-1c1e-40e6-994d-619340f782cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/898fcd26-1c1e-40e6-994d-619340f782cb-istiod-ca-cert\") pod \"898fcd26-1c1e-40e6-994d-619340f782cb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") "
Oct 30 19:08:59.626187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.625953    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-envoy\") pod \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") "
Oct 30 19:08:59.626187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.625984    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-envoy\") pod \"898fcd26-1c1e-40e6-994d-619340f782cb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") "
Oct 30 19:08:59.626187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.625999    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-credential-socket\") pod \"898fcd26-1c1e-40e6-994d-619340f782cb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") "
Oct 30 19:08:59.626821 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.626255    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-certs\") pod \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") "
Oct 30 19:08:59.626821 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.626292    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-credential-socket\") pod \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\" (UID: \"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0\") "
Oct 30 19:08:59.626821 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.626328    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-kube-api-access-kplx8\" (UniqueName: \"kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-kube-api-access-kplx8\") pod \"898fcd26-1c1e-40e6-994d-619340f782cb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") "
Oct 30 19:08:59.626821 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.626359    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/898fcd26-1c1e-40e6-994d-619340f782cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/898fcd26-1c1e-40e6-994d-619340f782cb-istio-podinfo\") pod \"898fcd26-1c1e-40e6-994d-619340f782cb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") "
Oct 30 19:08:59.626821 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.626375    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-istio-token\" (UniqueName: \"kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-istio-token\") pod \"898fcd26-1c1e-40e6-994d-619340f782cb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") "
Oct 30 19:08:59.626821 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.626417    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-data\") pod \"898fcd26-1c1e-40e6-994d-619340f782cb\" (UID: \"898fcd26-1c1e-40e6-994d-619340f782cb\") "
Oct 30 19:08:59.626821 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.626429    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-socket" pod "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" (UID: "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.626821 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.626641    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-socket" pod "898fcd26-1c1e-40e6-994d-619340f782cb" (UID: "898fcd26-1c1e-40e6-994d-619340f782cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.627528 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.626842    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-data" pod "898fcd26-1c1e-40e6-994d-619340f782cb" (UID: "898fcd26-1c1e-40e6-994d-619340f782cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.627528 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.627105    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-certs" pod "898fcd26-1c1e-40e6-994d-619340f782cb" (UID: "898fcd26-1c1e-40e6-994d-619340f782cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.627749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.627726    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-credential-socket" pod "898fcd26-1c1e-40e6-994d-619340f782cb" (UID: "898fcd26-1c1e-40e6-994d-619340f782cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.627788 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.627752    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.627788 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.627776    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.627858 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.627790    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.627858 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.627806    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.627858 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.627820    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.627858 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.627834    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.627858 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.627847    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.628320 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.627860    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9dqh\" (UniqueName: \"kubernetes.io/projected/58b2dba2-c676-4658-8dc3-d47d9782b180-kube-api-access-q9dqh\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.628320 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.627876    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b2dba2-c676-4658-8dc3-d47d9782b180-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.628320 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.628089    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-certs" pod "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" (UID: "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.628320 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.628296    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-credential-socket" pod "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" (UID: "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.629711 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.629116    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istiod-ca-cert" pod "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" (UID: "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:59.629711 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.629288    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898fcd26-1c1e-40e6-994d-619340f782cb-istiod-ca-cert" pod "898fcd26-1c1e-40e6-994d-619340f782cb" (UID: "898fcd26-1c1e-40e6-994d-619340f782cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:59.633357 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.633326    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-envoy" pod "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" (UID: "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.637712 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.637642    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-istio-token" pod "898fcd26-1c1e-40e6-994d-619340f782cb" (UID: "898fcd26-1c1e-40e6-994d-619340f782cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:59.637712 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.637683    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-kube-api-access-msphp" pod "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" (UID: "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0"). InnerVolumeSpecName "kube-api-access-msphp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:59.638075 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.637999    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/898fcd26-1c1e-40e6-994d-619340f782cb-istio-podinfo" pod "898fcd26-1c1e-40e6-994d-619340f782cb" (UID: "898fcd26-1c1e-40e6-994d-619340f782cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:08:59.638075 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.638034    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-kube-api-access-kplx8" pod "898fcd26-1c1e-40e6-994d-619340f782cb" (UID: "898fcd26-1c1e-40e6-994d-619340f782cb"). InnerVolumeSpecName "kube-api-access-kplx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:59.644373 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.643421    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-envoy" pod "898fcd26-1c1e-40e6-994d-619340f782cb" (UID: "898fcd26-1c1e-40e6-994d-619340f782cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:59.682184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.681991    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-podinfo" pod "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" (UID: "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:08:59.688496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.688443    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-token" pod "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" (UID: "ffd3e34f-96e3-4fee-9e15-4b884c33b0c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:59.728962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.728919    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/898fcd26-1c1e-40e6-994d-619340f782cb-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.728962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.728949    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.728962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.728959    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.728962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.728968    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/898fcd26-1c1e-40e6-994d-619340f782cb-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.728962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.728977    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.729424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.728985    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.729424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.728994    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kplx8\" (UniqueName: \"kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-kube-api-access-kplx8\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.729424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.729003    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/898fcd26-1c1e-40e6-994d-619340f782cb-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.729424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.729013    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/898fcd26-1c1e-40e6-994d-619340f782cb-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.729424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.729021    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.729424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.729028    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.729424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.729039    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.729424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.729052    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msphp\" (UniqueName: \"kubernetes.io/projected/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0-kube-api-access-msphp\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:59.791929 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.791897    6014 generic.go:356] "Generic (PLEG): container finished" podID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerID="3a67e298f1ea43f5a7100542fcbfef257a2602621e1c1bea56569811822761a5" exitCode=0
Oct 30 19:08:59.792093 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.791996    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" event={"ID":"58b2dba2-c676-4658-8dc3-d47d9782b180","Type":"ContainerDied","Data":"3a67e298f1ea43f5a7100542fcbfef257a2602621e1c1bea56569811822761a5"}
Oct 30 19:08:59.792093 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.792066    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn" event={"ID":"58b2dba2-c676-4658-8dc3-d47d9782b180","Type":"ContainerDied","Data":"e8a783c5a7e83eabca1393180399c7c8509e32d0eefc72d48ea1049f40633096"}
Oct 30 19:08:59.792093 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.792091    6014 scope.go:122] "RemoveContainer" containerID="3a67e298f1ea43f5a7100542fcbfef257a2602621e1c1bea56569811822761a5"
Oct 30 19:08:59.792262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.792248    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"
Oct 30 19:08:59.798418 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.798388    6014 generic.go:356] "Generic (PLEG): container finished" podID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerID="4640b3770dffad6254fb51f6b4ac62d506f1d92c9e5f105bdc00d2145aa8e026" exitCode=0
Oct 30 19:08:59.798577 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.798558    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" event={"ID":"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0","Type":"ContainerDied","Data":"4640b3770dffad6254fb51f6b4ac62d506f1d92c9e5f105bdc00d2145aa8e026"}
Oct 30 19:08:59.798628 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.798615    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l" event={"ID":"ffd3e34f-96e3-4fee-9e15-4b884c33b0c0","Type":"ContainerDied","Data":"6599d2c4981116a01a9f2e54ddfe6e8e808acc86b3718a064addcd9e86d360d8"}
Oct 30 19:08:59.798739 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.798727    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"
Oct 30 19:08:59.801269 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.801042    6014 generic.go:356] "Generic (PLEG): container finished" podID="898fcd26-1c1e-40e6-994d-619340f782cb" containerID="2f0282cc95f3af422d3a83d09c30f8f687312f19747accc2a3a8b495bfe36093" exitCode=0
Oct 30 19:08:59.801269 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.801106    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"
Oct 30 19:08:59.801269 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.801122    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" event={"ID":"898fcd26-1c1e-40e6-994d-619340f782cb","Type":"ContainerDied","Data":"2f0282cc95f3af422d3a83d09c30f8f687312f19747accc2a3a8b495bfe36093"}
Oct 30 19:08:59.801269 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.801187    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb" event={"ID":"898fcd26-1c1e-40e6-994d-619340f782cb","Type":"ContainerDied","Data":"225d702175cf3078b001dd02c68c9a80207daf421e97b6c1041d02ffcb67449e"}
Oct 30 19:08:59.873478 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.838180    6014 scope.go:122] "RemoveContainer" containerID="3a67e298f1ea43f5a7100542fcbfef257a2602621e1c1bea56569811822761a5"
Oct 30 19:08:59.873478 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:08:59.838554    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a67e298f1ea43f5a7100542fcbfef257a2602621e1c1bea56569811822761a5\": container with ID starting with 3a67e298f1ea43f5a7100542fcbfef257a2602621e1c1bea56569811822761a5 not found: ID does not exist" containerID="3a67e298f1ea43f5a7100542fcbfef257a2602621e1c1bea56569811822761a5"
Oct 30 19:08:59.873478 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.838585    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a67e298f1ea43f5a7100542fcbfef257a2602621e1c1bea56569811822761a5"} err="failed to get container status \"3a67e298f1ea43f5a7100542fcbfef257a2602621e1c1bea56569811822761a5\": rpc error: code = NotFound desc = could not find container \"3a67e298f1ea43f5a7100542fcbfef257a2602621e1c1bea56569811822761a5\": container with ID starting with 3a67e298f1ea43f5a7100542fcbfef257a2602621e1c1bea56569811822761a5 not found: ID does not exist"
Oct 30 19:08:59.873478 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.838607    6014 scope.go:122] "RemoveContainer" containerID="4640b3770dffad6254fb51f6b4ac62d506f1d92c9e5f105bdc00d2145aa8e026"
Oct 30 19:08:59.879474 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.878357    6014 scope.go:122] "RemoveContainer" containerID="4640b3770dffad6254fb51f6b4ac62d506f1d92c9e5f105bdc00d2145aa8e026"
Oct 30 19:08:59.879474 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:08:59.878843    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4640b3770dffad6254fb51f6b4ac62d506f1d92c9e5f105bdc00d2145aa8e026\": container with ID starting with 4640b3770dffad6254fb51f6b4ac62d506f1d92c9e5f105bdc00d2145aa8e026 not found: ID does not exist" containerID="4640b3770dffad6254fb51f6b4ac62d506f1d92c9e5f105bdc00d2145aa8e026"
Oct 30 19:08:59.879474 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.878885    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4640b3770dffad6254fb51f6b4ac62d506f1d92c9e5f105bdc00d2145aa8e026"} err="failed to get container status \"4640b3770dffad6254fb51f6b4ac62d506f1d92c9e5f105bdc00d2145aa8e026\": rpc error: code = NotFound desc = could not find container \"4640b3770dffad6254fb51f6b4ac62d506f1d92c9e5f105bdc00d2145aa8e026\": container with ID starting with 4640b3770dffad6254fb51f6b4ac62d506f1d92c9e5f105bdc00d2145aa8e026 not found: ID does not exist"
Oct 30 19:08:59.879474 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.878912    6014 scope.go:122] "RemoveContainer" containerID="2f0282cc95f3af422d3a83d09c30f8f687312f19747accc2a3a8b495bfe36093"
Oct 30 19:08:59.893906 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.893877    6014 scope.go:122] "RemoveContainer" containerID="2f0282cc95f3af422d3a83d09c30f8f687312f19747accc2a3a8b495bfe36093"
Oct 30 19:08:59.894329 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:08:59.894307    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0282cc95f3af422d3a83d09c30f8f687312f19747accc2a3a8b495bfe36093\": container with ID starting with 2f0282cc95f3af422d3a83d09c30f8f687312f19747accc2a3a8b495bfe36093 not found: ID does not exist" containerID="2f0282cc95f3af422d3a83d09c30f8f687312f19747accc2a3a8b495bfe36093"
Oct 30 19:08:59.894442 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.894336    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0282cc95f3af422d3a83d09c30f8f687312f19747accc2a3a8b495bfe36093"} err="failed to get container status \"2f0282cc95f3af422d3a83d09c30f8f687312f19747accc2a3a8b495bfe36093\": rpc error: code = NotFound desc = could not find container \"2f0282cc95f3af422d3a83d09c30f8f687312f19747accc2a3a8b495bfe36093\": container with ID starting with 2f0282cc95f3af422d3a83d09c30f8f687312f19747accc2a3a8b495bfe36093 not found: ID does not exist"
Oct 30 19:08:59.940459 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.940430    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"]
Oct 30 19:08:59.942509 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:08:59.942477    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85e7d51_316a_4e68_87e0_03217deade17.slice/crio-cf1ac3eedb59c58bc3982371782fe5de4c5a264aadfc57d7e0d7f0ad7ed7c207 WatchSource:0}: Error finding container cf1ac3eedb59c58bc3982371782fe5de4c5a264aadfc57d7e0d7f0ad7ed7c207: Status 404 returned error can't find the container with id cf1ac3eedb59c58bc3982371782fe5de4c5a264aadfc57d7e0d7f0ad7ed7c207
Oct 30 19:08:59.944173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:08:59.943702    6014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:09:00.076077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:00.076040    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"]
Oct 30 19:09:00.080169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:00.080127    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad6cf069-w9lgb"]
Oct 30 19:09:00.088957 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:00.088899    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"]
Oct 30 19:09:00.093339 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:00.093315    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-fp7rb"]
Oct 30 19:09:00.204732 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:00.204638    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"]
Oct 30 19:09:00.208988 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:00.208960    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-qsb6l"]
Oct 30 19:09:00.332377 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:00.332342    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"]
Oct 30 19:09:00.336013 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:00.335988    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-rxwbn"]
Oct 30 19:09:00.821808 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:00.818646    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9" event={"ID":"b85e7d51-316a-4e68-87e0-03217deade17","Type":"ContainerStarted","Data":"cf1ac3eedb59c58bc3982371782fe5de4c5a264aadfc57d7e0d7f0ad7ed7c207"}
Oct 30 19:09:01.482721 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:01.482690    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307904db-214a-46d3-9434-b7ece8f648c5" path="/var/lib/kubelet/pods/307904db-214a-46d3-9434-b7ece8f648c5/volumes"
Oct 30 19:09:01.483274 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:01.483259    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" path="/var/lib/kubelet/pods/58b2dba2-c676-4658-8dc3-d47d9782b180/volumes"
Oct 30 19:09:01.483750 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:01.483737    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="898fcd26-1c1e-40e6-994d-619340f782cb" path="/var/lib/kubelet/pods/898fcd26-1c1e-40e6-994d-619340f782cb/volumes"
Oct 30 19:09:01.484526 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:01.484511    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" path="/var/lib/kubelet/pods/ffd3e34f-96e3-4fee-9e15-4b884c33b0c0/volumes"
Oct 30 19:09:01.875184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:01.875133    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9" event={"ID":"b85e7d51-316a-4e68-87e0-03217deade17","Type":"ContainerStarted","Data":"c7876f441be1f72c928b416632c3e4054317349119f70445d2126507ec460bd3"}
Oct 30 19:09:01.875817 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:01.875797    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:09:01.993008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:01.992950    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9" podStartSLOduration=3.135397098 podStartE2EDuration="3.992933442s" totalImagesPullingTime="857.536344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:01.990023424 +0000 UTC m=+1780.633191039" watchObservedRunningTime="2026-10-30 19:09:01.992933442 +0000 UTC m=+1780.636101066"
Oct 30 19:09:07.927914 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:07.927871    6014 generic.go:356] "Generic (PLEG): container finished" podID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerID="2bbfb018dd3c8ce4fa048f873c3971203c5acac2502e45a81e7772984231cbe2" exitCode=0
Oct 30 19:09:07.928432 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:07.927934    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" event={"ID":"6d1c4684-874c-4d3d-aebb-517a892cb376","Type":"ContainerDied","Data":"2bbfb018dd3c8ce4fa048f873c3971203c5acac2502e45a81e7772984231cbe2"}
Oct 30 19:09:07.928493 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:07.928452    6014 scope.go:122] "RemoveContainer" containerID="2bbfb018dd3c8ce4fa048f873c3971203c5acac2502e45a81e7772984231cbe2"
Oct 30 19:09:07.930343 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:07.930313    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:09:07.930458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:07.930386    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:09:07.930458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:07.930421    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:09:08.897205 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:08.897173    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:09:08.938242 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:08.938207    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" event={"ID":"6d1c4684-874c-4d3d-aebb-517a892cb376","Type":"ContainerStarted","Data":"46283fd697700f9f52b70067fbcda31ca3db86b06fe0ead3384fbcc5b3df205f"}
Oct 30 19:09:09.244013 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:09.243925    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:09:09.244205 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:09.244177    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:09:09.244270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:09.244250    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 19:09:10.244022 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:10.243986    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:09:10.244480 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:10.244056    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 19:09:11.244056 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:11.244021    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:09:11.244546 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:11.244079    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 19:09:12.244597 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:12.244557    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:09:12.245031 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:12.244620    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 19:09:13.244404 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:13.244367    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:09:13.244598 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:13.244434    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 19:09:14.244353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:14.244303    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:09:14.244965 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:14.244392    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 19:09:15.059422 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:15.059378    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"]
Oct 30 19:09:15.108388 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:15.108354    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6355210-9wbqk"]
Oct 30 19:09:15.249279 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:15.249248    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:09:15.249750 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:15.249738    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:09:15.250449 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:15.250428    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:09:15.478709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:15.478675    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb4641dc-c870-4d9d-aee0-265967e8c5d3" path="/var/lib/kubelet/pods/cb4641dc-c870-4d9d-aee0-265967e8c5d3/volumes"
Oct 30 19:09:17.702998 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.702965    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"]
Oct 30 19:09:17.703816 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.703798    6014 cpu_manager.go:414] "containerMap: removing container" podUID="898fcd26-1c1e-40e6-994d-619340f782cb" containerName="istio-proxy"
Oct 30 19:09:17.703863 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.703818    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="898fcd26-1c1e-40e6-994d-619340f782cb" containerName="istio-proxy"
Oct 30 19:09:17.703863 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.703841    6014 cpu_manager.go:414] "containerMap: removing container" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy"
Oct 30 19:09:17.703863 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.703846    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy"
Oct 30 19:09:17.703962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.703877    6014 cpu_manager.go:414] "containerMap: removing container" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy"
Oct 30 19:09:17.703962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.703884    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy"
Oct 30 19:09:17.704031 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.703990    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b2dba2-c676-4658-8dc3-d47d9782b180" containerName="istio-proxy"
Oct 30 19:09:17.704031 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.704006    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd3e34f-96e3-4fee-9e15-4b884c33b0c0" containerName="istio-proxy"
Oct 30 19:09:17.704031 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.704018    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="898fcd26-1c1e-40e6-994d-619340f782cb" containerName="istio-proxy"
Oct 30 19:09:17.706848 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.706827    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.752857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.752816    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"]
Oct 30 19:09:17.756340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.756322    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.770239 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.770215    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-87ed12-openshift-default-dockercfg-f4qfc\""
Oct 30 19:09:17.788695 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.788645    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-certs\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.788880 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.788816    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-token\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.788944 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.788874    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-envoy\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.788992 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.788963    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.789047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.788998    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-certs\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.789047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.789030    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-data\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.789202 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.789178    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnx2n\" (UniqueName: \"kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-kube-api-access-xnx2n\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.789283 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.789246    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-envoy\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.789347 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.789314    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-credential-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.789402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.789350    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111205d6-93ca-400e-8383-1b1c555d47dc-istio-podinfo\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.789460 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.789406    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dfe5333-8253-43be-b7fe-c548ff82067c-istiod-ca-cert\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.790223 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.790199    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-podinfo\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.790350 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.790268    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sr9r\" (UniqueName: \"kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-kube-api-access-9sr9r\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.790402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.790359    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.790402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.790394    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-credential-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.790499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.790430    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-istio-token\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.790499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.790479    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111205d6-93ca-400e-8383-1b1c555d47dc-istiod-ca-cert\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.790607 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.790592    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-data\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.805641 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.805608    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"]
Oct 30 19:09:17.841916 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.841875    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"]
Oct 30 19:09:17.891815 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.891768    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-token\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.892008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.891839    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-envoy\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.892008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.891871    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.892008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.891891    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-certs\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.892008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.891916    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-data\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.892008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.891958    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnx2n\" (UniqueName: \"kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-kube-api-access-xnx2n\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.892363 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892093    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-envoy\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.892363 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892186    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-credential-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.892363 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892225    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111205d6-93ca-400e-8383-1b1c555d47dc-istio-podinfo\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.892363 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892269    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dfe5333-8253-43be-b7fe-c548ff82067c-istiod-ca-cert\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.892363 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892302    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-podinfo\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.892622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892374    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sr9r\" (UniqueName: \"kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-kube-api-access-9sr9r\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.892622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892438    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.892622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892442    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-data\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.892622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892466    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-certs\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.892622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892484    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-credential-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.892622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892536    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-credential-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.892622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892550    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-istio-token\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.892622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892441    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.892622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892583    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111205d6-93ca-400e-8383-1b1c555d47dc-istiod-ca-cert\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.893305 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892663    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-data\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.893305 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892713    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-certs\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.893305 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.892741    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-credential-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.893305 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.893230    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.893542 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.893319    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dfe5333-8253-43be-b7fe-c548ff82067c-istiod-ca-cert\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.893542 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.893409    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-data\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.893660 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.893580    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-certs\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.893660 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.893602    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111205d6-93ca-400e-8383-1b1c555d47dc-istiod-ca-cert\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.894727 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.894703    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-envoy\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.894727 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.894722    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-envoy\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.895046 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.895025    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111205d6-93ca-400e-8383-1b1c555d47dc-istio-podinfo\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.895587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.895570    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-podinfo\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.952532 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.952498    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sr9r\" (UniqueName: \"kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-kube-api-access-9sr9r\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:17.952690 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.952662    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnx2n\" (UniqueName: \"kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-kube-api-access-xnx2n\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.953267 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.953206    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-token\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:17.953607 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:17.953589    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-istio-token\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:18.023263 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:18.023227    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:18.070451 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:18.070414    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:18.458395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:18.458365    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"]
Oct 30 19:09:18.458881 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:09:18.458854    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfe5333_8253_43be_b7fe_c548ff82067c.slice/crio-38ede00d5c90a85aeaddc53184e298f2a984dd4a6dc079cafaef73cff3010975 WatchSource:0}: Error finding container 38ede00d5c90a85aeaddc53184e298f2a984dd4a6dc079cafaef73cff3010975: Status 404 returned error can't find the container with id 38ede00d5c90a85aeaddc53184e298f2a984dd4a6dc079cafaef73cff3010975
Oct 30 19:09:18.461823 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:18.461786    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:09:18.461926 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:18.461869    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:09:18.461926 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:18.461914    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:09:18.550409 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:18.550383    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"]
Oct 30 19:09:18.552365 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:09:18.552339    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111205d6_93ca_400e_8383_1b1c555d47dc.slice/crio-9236043218b5cd85596c54098d29cf203a29b18cf92e12504f7ce4fa31c13951 WatchSource:0}: Error finding container 9236043218b5cd85596c54098d29cf203a29b18cf92e12504f7ce4fa31c13951: Status 404 returned error can't find the container with id 9236043218b5cd85596c54098d29cf203a29b18cf92e12504f7ce4fa31c13951
Oct 30 19:09:18.555201 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:18.555171    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:09:18.555282 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:18.555235    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:09:18.555282 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:18.555265    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:09:19.057164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:19.048233    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5" event={"ID":"111205d6-93ca-400e-8383-1b1c555d47dc","Type":"ContainerStarted","Data":"434ae9326b9232252ee465690ad10e392ca7c3488607dd8e88d0b4c7cb1d7c8e"}
Oct 30 19:09:19.057164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:19.048304    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5" event={"ID":"111205d6-93ca-400e-8383-1b1c555d47dc","Type":"ContainerStarted","Data":"9236043218b5cd85596c54098d29cf203a29b18cf92e12504f7ce4fa31c13951"}
Oct 30 19:09:19.057734 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:19.057292    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6" event={"ID":"1dfe5333-8253-43be-b7fe-c548ff82067c","Type":"ContainerStarted","Data":"77e639fe12a0ff3936ade9bfea43b08746d3822bd1033f80091dd4ee2d68eed6"}
Oct 30 19:09:19.057734 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:19.057363    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6" event={"ID":"1dfe5333-8253-43be-b7fe-c548ff82067c","Type":"ContainerStarted","Data":"38ede00d5c90a85aeaddc53184e298f2a984dd4a6dc079cafaef73cff3010975"}
Oct 30 19:09:19.073690 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:19.071747    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:19.076312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:19.076261    6014 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:09:19.076510 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:19.076335    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5" podUID="111205d6-93ca-400e-8383-1b1c555d47dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 30 19:09:19.224617 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:19.224557    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5" podStartSLOduration=2.224540698 podStartE2EDuration="2.224540698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:19.219161465 +0000 UTC m=+1797.862329075" watchObservedRunningTime="2026-10-30 19:09:19.224540698 +0000 UTC m=+1797.867708312"
Oct 30 19:09:19.376166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:19.368322    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6" podStartSLOduration=2.368297868 podStartE2EDuration="2.368297868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:19.347658663 +0000 UTC m=+1797.990826289" watchObservedRunningTime="2026-10-30 19:09:19.368297868 +0000 UTC m=+1798.011465483"
Oct 30 19:09:20.025187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:20.025121    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:20.027419 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:20.026421    6014 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:09:20.027419 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:20.026481    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6" podUID="1dfe5333-8253-43be-b7fe-c548ff82067c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 19:09:20.086166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:20.084562    6014 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:09:20.086166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:20.084625    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5" podUID="111205d6-93ca-400e-8383-1b1c555d47dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 30 19:09:21.024297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:21.024259    6014 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:09:21.024480 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:21.024321    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6" podUID="1dfe5333-8253-43be-b7fe-c548ff82067c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 19:09:21.073581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:21.073541    6014 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:09:21.073842 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:21.073811    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5" podUID="111205d6-93ca-400e-8383-1b1c555d47dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 30 19:09:22.024166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:22.024114    6014 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:09:22.024655 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:22.024189    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6" podUID="1dfe5333-8253-43be-b7fe-c548ff82067c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 19:09:22.081165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:22.077515    6014 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:09:22.081165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:22.077575    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5" podUID="111205d6-93ca-400e-8383-1b1c555d47dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 30 19:09:23.024166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:23.024065    6014 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:09:23.024166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:23.024116    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6" podUID="1dfe5333-8253-43be-b7fe-c548ff82067c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 19:09:23.088166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:23.075438    6014 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:09:23.088166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:23.075516    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5" podUID="111205d6-93ca-400e-8383-1b1c555d47dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 30 19:09:23.306731 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:09:23.306636    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:09:23.910024 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:23.907473    6014 scope.go:122] "RemoveContainer" containerID="23c7315790873256077b4d593263230fdbbaddea0a0b85b3303d347d8ddbae3a"
Oct 30 19:09:23.920057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:23.920035    6014 scope.go:122] "RemoveContainer" containerID="004d3304ffbc8b80cba400848d4e93d5e22fe38a2737c7f6a4c7cb849dafafb3"
Oct 30 19:09:23.930611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:23.930589    6014 scope.go:122] "RemoveContainer" containerID="15e038be7f76c396248c41fa1f3138a5690e3735ed929ba536f83f857cf05af1"
Oct 30 19:09:24.023625 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:24.023582    6014 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:09:24.023724 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:24.023661    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6" podUID="1dfe5333-8253-43be-b7fe-c548ff82067c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 19:09:24.103209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:24.100193    6014 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:09:24.104240 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:24.104166    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5" podUID="111205d6-93ca-400e-8383-1b1c555d47dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 30 19:09:24.605807 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:24.605762    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"]
Oct 30 19:09:24.606233 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:24.606007    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" podUID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" containerName="istio-proxy" containerID="cri-o://f56562b375c2303dc0297bf5f0f3ac4a84fac7e52ed8a64731ca24513f9e29f6" gracePeriod=30
Oct 30 19:09:24.611828 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:24.611807    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"]
Oct 30 19:09:24.612019 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:24.612000    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy" containerID="cri-o://686b6a276fd33a82fd58defbd76458c55c1cbbfd8a0b12d55ac2d82f3aabbd2e" gracePeriod=30
Oct 30 19:09:24.617286 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:24.617237    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"]
Oct 30 19:09:24.617478 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:24.617447    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" podUID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" containerName="istio-proxy" containerID="cri-o://7dd0d4e3f6702f579db8bc8f143cd3162712cf8a74c267328e83f4e55f467192" gracePeriod=30
Oct 30 19:09:24.679107 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:24.679053    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"]
Oct 30 19:09:24.680557 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:24.680488    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh" podUID="a62c6f22-f5b9-4061-9a75-ea765cf1a199" containerName="istio-proxy" containerID="cri-o://c98c060f935cdfef02c3a8a4cfa642eeb5cd28f508bbd14b78d0b3945a0c2e9e" gracePeriod=30
Oct 30 19:09:25.028904 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:25.028826    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:25.029179 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:25.029111    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:25.029770 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:25.029753    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:09:25.076309 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:25.076276    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:25.076629 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:25.076598    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:25.077308 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:25.077289    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:09:26.717423 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:26.717388    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:09:26.717810 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:26.717457    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" podUID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:09:28.705909 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:28.705838    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:09:28.705909 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:28.705894    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh" podUID="a62c6f22-f5b9-4061-9a75-ea765cf1a199" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:09:29.286859 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.286815    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"]
Oct 30 19:09:29.293800 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.293780    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.382743 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.382703    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f081490d-ac76-40cc-9df9-14aafe2726ad-istiod-ca-cert\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.383205 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.383177    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-data\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.383468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.383446    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-token\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.383730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.383708    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-credential-socket\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.383921 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.383902    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-envoy\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.384112 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.384092    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-certs\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.384469 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.384446    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-podinfo\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.384664 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.384641    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-socket\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.385462 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.385430    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd595\" (UniqueName: \"kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-kube-api-access-nd595\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.400741 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.400691    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"]
Oct 30 19:09:29.540491 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.540394    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2e004a-openshift-default-dockercfg-79zh9\""
Oct 30 19:09:29.541953 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.541449    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd595\" (UniqueName: \"kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-kube-api-access-nd595\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.544355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.542323    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f081490d-ac76-40cc-9df9-14aafe2726ad-istiod-ca-cert\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.544355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.542409    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-data\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.544355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.542437    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-token\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.544355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.542520    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-credential-socket\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.544355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.542547    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-envoy\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.544355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.542579    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-certs\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.544355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.542724    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-podinfo\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.544355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.542754    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-socket\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.544355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.543302    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-socket\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.544355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.543932    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f081490d-ac76-40cc-9df9-14aafe2726ad-istiod-ca-cert\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.544987 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.544469    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-data\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.549291 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.545077    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-credential-socket\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.549291 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.546500    6014 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:09:29.549291 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.546563    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" podUID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:09:29.549291 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.547687    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-certs\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.549291 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.548090    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-envoy\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.565267 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.565204    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-podinfo\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.601187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.601103    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd595\" (UniqueName: \"kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-kube-api-access-nd595\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.602051 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.602031    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-token\") pod \"e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") " pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:29.617634 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:29.617532    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:30.031911 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.031878    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:09:30.129716 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.129681    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"]
Oct 30 19:09:30.130689 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.130639    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:09:30.135586 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:09:30.135556    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf081490d_ac76_40cc_9df9_14aafe2726ad.slice/crio-b0c159af3c533ff753998c6a76fa4f1d468f9698c895f2ed05deb0c454816583 WatchSource:0}: Error finding container b0c159af3c533ff753998c6a76fa4f1d468f9698c895f2ed05deb0c454816583: Status 404 returned error can't find the container with id b0c159af3c533ff753998c6a76fa4f1d468f9698c895f2ed05deb0c454816583
Oct 30 19:09:30.136395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.136367    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:09:30.144563 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.144511    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:09:30.144699 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.144603    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:09:30.144699 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.144648    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:09:30.148223 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.148051    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 19:09:30.159783 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.159760    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istiod-ca-cert\") pod \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") "
Oct 30 19:09:30.159888 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.159823    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-kube-api-access-7q27x\" (UniqueName: \"kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-kube-api-access-7q27x\") pod \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") "
Oct 30 19:09:30.159888 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.159876    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-podinfo\") pod \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") "
Oct 30 19:09:30.160014 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.159995    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-data\") pod \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") "
Oct 30 19:09:30.160087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.160038    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-certs\") pod \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") "
Oct 30 19:09:30.160087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.160083    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-envoy\") pod \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") "
Oct 30 19:09:30.160231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.160165    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istiod-ca-cert" pod "a62c6f22-f5b9-4061-9a75-ea765cf1a199" (UID: "a62c6f22-f5b9-4061-9a75-ea765cf1a199"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:30.160276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.160235    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-token\" (UniqueName: \"kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-token\") pod \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") "
Oct 30 19:09:30.160333 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.160273    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-credential-socket\") pod \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") "
Oct 30 19:09:30.160385 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.160363    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-data" pod "a62c6f22-f5b9-4061-9a75-ea765cf1a199" (UID: "a62c6f22-f5b9-4061-9a75-ea765cf1a199"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.160446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.160421    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-socket\") pod \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\" (UID: \"a62c6f22-f5b9-4061-9a75-ea765cf1a199\") "
Oct 30 19:09:30.160446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.160430    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-certs" pod "a62c6f22-f5b9-4061-9a75-ea765cf1a199" (UID: "a62c6f22-f5b9-4061-9a75-ea765cf1a199"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.160769 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.160688    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-credential-socket" pod "a62c6f22-f5b9-4061-9a75-ea765cf1a199" (UID: "a62c6f22-f5b9-4061-9a75-ea765cf1a199"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.160864 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.160725    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-socket" pod "a62c6f22-f5b9-4061-9a75-ea765cf1a199" (UID: "a62c6f22-f5b9-4061-9a75-ea765cf1a199"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.161255 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.161227    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.161354 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.161256    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.161354 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.161271    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.161354 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.161284    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.161354 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.161296    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.162388 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.162362    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-podinfo" pod "a62c6f22-f5b9-4061-9a75-ea765cf1a199" (UID: "a62c6f22-f5b9-4061-9a75-ea765cf1a199"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:30.162899 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.162820    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-kube-api-access-7q27x" pod "a62c6f22-f5b9-4061-9a75-ea765cf1a199" (UID: "a62c6f22-f5b9-4061-9a75-ea765cf1a199"). InnerVolumeSpecName "kube-api-access-7q27x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:30.162899 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.162847    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-envoy" pod "a62c6f22-f5b9-4061-9a75-ea765cf1a199" (UID: "a62c6f22-f5b9-4061-9a75-ea765cf1a199"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.163316 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.163300    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-token" pod "a62c6f22-f5b9-4061-9a75-ea765cf1a199" (UID: "a62c6f22-f5b9-4061-9a75-ea765cf1a199"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:30.261838 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.261802    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-socket\") pod \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") "
Oct 30 19:09:30.262041 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.261886    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-envoy\") pod \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") "
Oct 30 19:09:30.262041 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.261930    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-data\") pod \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") "
Oct 30 19:09:30.262041 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.261959    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-data\") pod \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") "
Oct 30 19:09:30.262041 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262006    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-credential-socket\") pod \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") "
Oct 30 19:09:30.262312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262070    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-data\") pod \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") "
Oct 30 19:09:30.262312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262098    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-envoy\") pod \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") "
Oct 30 19:09:30.262312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262197    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-socket\") pod \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") "
Oct 30 19:09:30.262312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262251    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-certs\") pod \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") "
Oct 30 19:09:30.262312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262297    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-podinfo\") pod \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") "
Oct 30 19:09:30.262575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262330    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-token\" (UniqueName: \"kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-token\") pod \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") "
Oct 30 19:09:30.262575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262374    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-kube-api-access-2lwwz\" (UniqueName: \"kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-kube-api-access-2lwwz\") pod \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") "
Oct 30 19:09:30.262575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262416    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-credential-socket\") pod \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") "
Oct 30 19:09:30.262575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262510    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-certs\") pod \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") "
Oct 30 19:09:30.262765 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262619    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-podinfo\") pod \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") "
Oct 30 19:09:30.262765 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262650    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-token\" (UniqueName: \"kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-token\") pod \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") "
Oct 30 19:09:30.262765 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262720    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-envoy\") pod \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") "
Oct 30 19:09:30.262918 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262782    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istiod-ca-cert\") pod \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") "
Oct 30 19:09:30.262918 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262841    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-kube-api-access-4qtbj\" (UniqueName: \"kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-kube-api-access-4qtbj\") pod \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") "
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262936    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-certs\") pod \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") "
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.262971    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istiod-ca-cert\") pod \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\" (UID: \"99ebfc68-3553-4bdb-9393-bc3b582c0bc0\") "
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.263028    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-kube-api-access-6kzbc\" (UniqueName: \"kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-kube-api-access-6kzbc\") pod \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") "
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.263060    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-token\" (UniqueName: \"kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-token\") pod \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") "
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.263089    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-socket\") pod \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") "
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.263160    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-podinfo\") pod \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\" (UID: \"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed\") "
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.263198    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istiod-ca-cert\") pod \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") "
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.263269    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-credential-socket\") pod \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\" (UID: \"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2\") "
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.264235    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q27x\" (UniqueName: \"kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-kube-api-access-7q27x\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.264261    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.264277    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.264291    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62c6f22-f5b9-4061-9a75-ea765cf1a199-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.264530    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-credential-socket" pod "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" (UID: "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.264871    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istiod-ca-cert" pod "99ebfc68-3553-4bdb-9393-bc3b582c0bc0" (UID: "99ebfc68-3553-4bdb-9393-bc3b582c0bc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.265093    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-socket" pod "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" (UID: "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.265412    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-certs" pod "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" (UID: "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.265690    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-envoy" pod "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" (UID: "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.265862    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istiod-ca-cert" pod "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" (UID: "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.266084    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-socket" pod "99ebfc68-3553-4bdb-9393-bc3b582c0bc0" (UID: "99ebfc68-3553-4bdb-9393-bc3b582c0bc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.266325    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-data" pod "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" (UID: "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.266493    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-data" pod "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" (UID: "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.266661    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-credential-socket" pod "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" (UID: "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.266868    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-data" pod "99ebfc68-3553-4bdb-9393-bc3b582c0bc0" (UID: "99ebfc68-3553-4bdb-9393-bc3b582c0bc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.267131    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-certs" pod "99ebfc68-3553-4bdb-9393-bc3b582c0bc0" (UID: "99ebfc68-3553-4bdb-9393-bc3b582c0bc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.267329    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-certs" pod "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" (UID: "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.269203    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-envoy" pod "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" (UID: "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.270401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.269961    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-credential-socket" pod "99ebfc68-3553-4bdb-9393-bc3b582c0bc0" (UID: "99ebfc68-3553-4bdb-9393-bc3b582c0bc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.271871 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.270532    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-kube-api-access-2lwwz" pod "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" (UID: "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed"). InnerVolumeSpecName "kube-api-access-2lwwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:30.271871 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.270554    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-token" pod "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" (UID: "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:30.271871 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.270744    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-socket" pod "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" (UID: "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.271871 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.270830    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istiod-ca-cert" pod "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" (UID: "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:30.271871 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.271522    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-envoy" pod "99ebfc68-3553-4bdb-9393-bc3b582c0bc0" (UID: "99ebfc68-3553-4bdb-9393-bc3b582c0bc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:30.272359 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.272322    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-podinfo" pod "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" (UID: "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:30.273203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.272598    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-token" pod "99ebfc68-3553-4bdb-9393-bc3b582c0bc0" (UID: "99ebfc68-3553-4bdb-9393-bc3b582c0bc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:30.273203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.272632    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-kube-api-access-6kzbc" pod "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" (UID: "d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2"). InnerVolumeSpecName "kube-api-access-6kzbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:30.273203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.272720    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-kube-api-access-4qtbj" pod "99ebfc68-3553-4bdb-9393-bc3b582c0bc0" (UID: "99ebfc68-3553-4bdb-9393-bc3b582c0bc0"). InnerVolumeSpecName "kube-api-access-4qtbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:30.273203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.273159    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-token" pod "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" (UID: "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:30.273543 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.273527    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-podinfo" pod "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" (UID: "7f2c9ad2-ed90-4a65-adcf-3b2540f128ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:30.273934 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.273912    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-podinfo" pod "99ebfc68-3553-4bdb-9393-bc3b582c0bc0" (UID: "99ebfc68-3553-4bdb-9393-bc3b582c0bc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:30.274674 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.274648    6014 generic.go:356] "Generic (PLEG): container finished" podID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" containerID="7dd0d4e3f6702f579db8bc8f143cd3162712cf8a74c267328e83f4e55f467192" exitCode=0
Oct 30 19:09:30.274779 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.274704    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"
Oct 30 19:09:30.274779 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.274742    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" event={"ID":"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed","Type":"ContainerDied","Data":"7dd0d4e3f6702f579db8bc8f143cd3162712cf8a74c267328e83f4e55f467192"}
Oct 30 19:09:30.274882 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.274793    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk" event={"ID":"7f2c9ad2-ed90-4a65-adcf-3b2540f128ed","Type":"ContainerDied","Data":"bc6d85dd84d6b03e670f3f79d9de4acf02d8fa47594dc3df7388c5950940647e"}
Oct 30 19:09:30.274882 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.274815    6014 scope.go:122] "RemoveContainer" containerID="7dd0d4e3f6702f579db8bc8f143cd3162712cf8a74c267328e83f4e55f467192"
Oct 30 19:09:30.277289 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.277262    6014 generic.go:356] "Generic (PLEG): container finished" podID="a62c6f22-f5b9-4061-9a75-ea765cf1a199" containerID="c98c060f935cdfef02c3a8a4cfa642eeb5cd28f508bbd14b78d0b3945a0c2e9e" exitCode=0
Oct 30 19:09:30.277412 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.277314    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"
Oct 30 19:09:30.277412 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.277354    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh" event={"ID":"a62c6f22-f5b9-4061-9a75-ea765cf1a199","Type":"ContainerDied","Data":"c98c060f935cdfef02c3a8a4cfa642eeb5cd28f508bbd14b78d0b3945a0c2e9e"}
Oct 30 19:09:30.277412 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.277408    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh" event={"ID":"a62c6f22-f5b9-4061-9a75-ea765cf1a199","Type":"ContainerDied","Data":"67d6f9b6c0d3ed6b6fd5ac08a3f96e2a69be612470a0db2d60e9469200aa1f15"}
Oct 30 19:09:30.284596 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.282446    6014 generic.go:356] "Generic (PLEG): container finished" podID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerID="686b6a276fd33a82fd58defbd76458c55c1cbbfd8a0b12d55ac2d82f3aabbd2e" exitCode=0
Oct 30 19:09:30.284596 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.282508    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"
Oct 30 19:09:30.284596 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.282529    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" event={"ID":"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2","Type":"ContainerDied","Data":"686b6a276fd33a82fd58defbd76458c55c1cbbfd8a0b12d55ac2d82f3aabbd2e"}
Oct 30 19:09:30.284596 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.282573    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29" event={"ID":"d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2","Type":"ContainerDied","Data":"36dbf4d17181684caa9cac3f9189278503720b2e744985f54f15779dbce3883c"}
Oct 30 19:09:30.284855 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.284754    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk" event={"ID":"f081490d-ac76-40cc-9df9-14aafe2726ad","Type":"ContainerStarted","Data":"cd629341e9b682c67cb34fdec95952f1a47018a6dd68c251c594f7e6ba365728"}
Oct 30 19:09:30.284855 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.284794    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk" event={"ID":"f081490d-ac76-40cc-9df9-14aafe2726ad","Type":"ContainerStarted","Data":"b0c159af3c533ff753998c6a76fa4f1d468f9698c895f2ed05deb0c454816583"}
Oct 30 19:09:30.287076 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.287053    6014 generic.go:356] "Generic (PLEG): container finished" podID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" containerID="f56562b375c2303dc0297bf5f0f3ac4a84fac7e52ed8a64731ca24513f9e29f6" exitCode=0
Oct 30 19:09:30.287241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.287085    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" event={"ID":"99ebfc68-3553-4bdb-9393-bc3b582c0bc0","Type":"ContainerDied","Data":"f56562b375c2303dc0297bf5f0f3ac4a84fac7e52ed8a64731ca24513f9e29f6"}
Oct 30 19:09:30.287241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.287119    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q" event={"ID":"99ebfc68-3553-4bdb-9393-bc3b582c0bc0","Type":"ContainerDied","Data":"a47695b7ae602d80419ff673c57ed0488b33313988354db0e470b6e2d4ca813a"}
Oct 30 19:09:30.287241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.287198    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"
Oct 30 19:09:30.288760 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.288742    6014 scope.go:122] "RemoveContainer" containerID="7dd0d4e3f6702f579db8bc8f143cd3162712cf8a74c267328e83f4e55f467192"
Oct 30 19:09:30.289062 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:09:30.289038    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd0d4e3f6702f579db8bc8f143cd3162712cf8a74c267328e83f4e55f467192\": container with ID starting with 7dd0d4e3f6702f579db8bc8f143cd3162712cf8a74c267328e83f4e55f467192 not found: ID does not exist" containerID="7dd0d4e3f6702f579db8bc8f143cd3162712cf8a74c267328e83f4e55f467192"
Oct 30 19:09:30.289128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.289077    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd0d4e3f6702f579db8bc8f143cd3162712cf8a74c267328e83f4e55f467192"} err="failed to get container status \"7dd0d4e3f6702f579db8bc8f143cd3162712cf8a74c267328e83f4e55f467192\": rpc error: code = NotFound desc = could not find container \"7dd0d4e3f6702f579db8bc8f143cd3162712cf8a74c267328e83f4e55f467192\": container with ID starting with 7dd0d4e3f6702f579db8bc8f143cd3162712cf8a74c267328e83f4e55f467192 not found: ID does not exist"
Oct 30 19:09:30.289128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.289100    6014 scope.go:122] "RemoveContainer" containerID="c98c060f935cdfef02c3a8a4cfa642eeb5cd28f508bbd14b78d0b3945a0c2e9e"
Oct 30 19:09:30.304308 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.304285    6014 scope.go:122] "RemoveContainer" containerID="c98c060f935cdfef02c3a8a4cfa642eeb5cd28f508bbd14b78d0b3945a0c2e9e"
Oct 30 19:09:30.304622 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:09:30.304597    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98c060f935cdfef02c3a8a4cfa642eeb5cd28f508bbd14b78d0b3945a0c2e9e\": container with ID starting with c98c060f935cdfef02c3a8a4cfa642eeb5cd28f508bbd14b78d0b3945a0c2e9e not found: ID does not exist" containerID="c98c060f935cdfef02c3a8a4cfa642eeb5cd28f508bbd14b78d0b3945a0c2e9e"
Oct 30 19:09:30.304660 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.304637    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98c060f935cdfef02c3a8a4cfa642eeb5cd28f508bbd14b78d0b3945a0c2e9e"} err="failed to get container status \"c98c060f935cdfef02c3a8a4cfa642eeb5cd28f508bbd14b78d0b3945a0c2e9e\": rpc error: code = NotFound desc = could not find container \"c98c060f935cdfef02c3a8a4cfa642eeb5cd28f508bbd14b78d0b3945a0c2e9e\": container with ID starting with c98c060f935cdfef02c3a8a4cfa642eeb5cd28f508bbd14b78d0b3945a0c2e9e not found: ID does not exist"
Oct 30 19:09:30.304660 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.304655    6014 scope.go:122] "RemoveContainer" containerID="686b6a276fd33a82fd58defbd76458c55c1cbbfd8a0b12d55ac2d82f3aabbd2e"
Oct 30 19:09:30.318913 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.318885    6014 scope.go:122] "RemoveContainer" containerID="686b6a276fd33a82fd58defbd76458c55c1cbbfd8a0b12d55ac2d82f3aabbd2e"
Oct 30 19:09:30.319340 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:09:30.319316    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686b6a276fd33a82fd58defbd76458c55c1cbbfd8a0b12d55ac2d82f3aabbd2e\": container with ID starting with 686b6a276fd33a82fd58defbd76458c55c1cbbfd8a0b12d55ac2d82f3aabbd2e not found: ID does not exist" containerID="686b6a276fd33a82fd58defbd76458c55c1cbbfd8a0b12d55ac2d82f3aabbd2e"
Oct 30 19:09:30.319453 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.319351    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686b6a276fd33a82fd58defbd76458c55c1cbbfd8a0b12d55ac2d82f3aabbd2e"} err="failed to get container status \"686b6a276fd33a82fd58defbd76458c55c1cbbfd8a0b12d55ac2d82f3aabbd2e\": rpc error: code = NotFound desc = could not find container \"686b6a276fd33a82fd58defbd76458c55c1cbbfd8a0b12d55ac2d82f3aabbd2e\": container with ID starting with 686b6a276fd33a82fd58defbd76458c55c1cbbfd8a0b12d55ac2d82f3aabbd2e not found: ID does not exist"
Oct 30 19:09:30.319453 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.319370    6014 scope.go:122] "RemoveContainer" containerID="f56562b375c2303dc0297bf5f0f3ac4a84fac7e52ed8a64731ca24513f9e29f6"
Oct 30 19:09:30.330510 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.330487    6014 scope.go:122] "RemoveContainer" containerID="f56562b375c2303dc0297bf5f0f3ac4a84fac7e52ed8a64731ca24513f9e29f6"
Oct 30 19:09:30.332168 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:09:30.331463    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56562b375c2303dc0297bf5f0f3ac4a84fac7e52ed8a64731ca24513f9e29f6\": container with ID starting with f56562b375c2303dc0297bf5f0f3ac4a84fac7e52ed8a64731ca24513f9e29f6 not found: ID does not exist" containerID="f56562b375c2303dc0297bf5f0f3ac4a84fac7e52ed8a64731ca24513f9e29f6"
Oct 30 19:09:30.332168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.331503    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56562b375c2303dc0297bf5f0f3ac4a84fac7e52ed8a64731ca24513f9e29f6"} err="failed to get container status \"f56562b375c2303dc0297bf5f0f3ac4a84fac7e52ed8a64731ca24513f9e29f6\": rpc error: code = NotFound desc = could not find container \"f56562b375c2303dc0297bf5f0f3ac4a84fac7e52ed8a64731ca24513f9e29f6\": container with ID starting with f56562b375c2303dc0297bf5f0f3ac4a84fac7e52ed8a64731ca24513f9e29f6 not found: ID does not exist"
Oct 30 19:09:30.366464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366379    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.366464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366406    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.366464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366417    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.366464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366428    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.366464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366438    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.366464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366447    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.366464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366455    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.366464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366463    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.366464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366472    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366481    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366489    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366498    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lwwz\" (UniqueName: \"kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-kube-api-access-2lwwz\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366507    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366515    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366523    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366533    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366544    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366553    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366561    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qtbj\" (UniqueName: \"kubernetes.io/projected/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-kube-api-access-4qtbj\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366569    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366577    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ebfc68-3553-4bdb-9393-bc3b582c0bc0-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366586    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzbc\" (UniqueName: \"kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-kube-api-access-6kzbc\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366594    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366603    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366611    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366619    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.367065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.366627    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:30.618645 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.618509    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:30.708509 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.708476    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"]
Oct 30 19:09:30.712196 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.712174    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-6495q"]
Oct 30 19:09:30.831834 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.831777    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk" podStartSLOduration=1.831762296 podStartE2EDuration="1.831762296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:30.827532743 +0000 UTC m=+1809.470700366" watchObservedRunningTime="2026-10-30 19:09:30.831762296 +0000 UTC m=+1809.474929910"
Oct 30 19:09:30.945287 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.945203    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"]
Oct 30 19:09:30.952713 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:30.952666    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-k48gk"]
Oct 30 19:09:31.080058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:31.080013    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"]
Oct 30 19:09:31.090321 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:31.090290    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-n46rh"]
Oct 30 19:09:31.213618 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:31.213513    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"]
Oct 30 19:09:31.218670 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:31.218639    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-t8s29"]
Oct 30 19:09:31.478176 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:31.478081    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" path="/var/lib/kubelet/pods/7f2c9ad2-ed90-4a65-adcf-3b2540f128ed/volumes"
Oct 30 19:09:31.478656 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:31.478641    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" path="/var/lib/kubelet/pods/99ebfc68-3553-4bdb-9393-bc3b582c0bc0/volumes"
Oct 30 19:09:31.479075 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:31.479062    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62c6f22-f5b9-4061-9a75-ea765cf1a199" path="/var/lib/kubelet/pods/a62c6f22-f5b9-4061-9a75-ea765cf1a199/volumes"
Oct 30 19:09:31.479876 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:31.479860    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" path="/var/lib/kubelet/pods/d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2/volumes"
Oct 30 19:09:31.624585 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:31.624558    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:32.307796 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:32.307763    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:32.308757 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:32.308739    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:09:35.186734 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:35.186694    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:09:37.541162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.538552    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"]
Oct 30 19:09:37.541162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.539840    6014 cpu_manager.go:414] "containerMap: removing container" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy"
Oct 30 19:09:37.541162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.539862    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy"
Oct 30 19:09:37.541162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.539879    6014 cpu_manager.go:414] "containerMap: removing container" podUID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" containerName="istio-proxy"
Oct 30 19:09:37.541162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.539887    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" containerName="istio-proxy"
Oct 30 19:09:37.541162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.539905    6014 cpu_manager.go:414] "containerMap: removing container" podUID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" containerName="istio-proxy"
Oct 30 19:09:37.541162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.539913    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" containerName="istio-proxy"
Oct 30 19:09:37.541162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.539929    6014 cpu_manager.go:414] "containerMap: removing container" podUID="a62c6f22-f5b9-4061-9a75-ea765cf1a199" containerName="istio-proxy"
Oct 30 19:09:37.541162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.539936    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62c6f22-f5b9-4061-9a75-ea765cf1a199" containerName="istio-proxy"
Oct 30 19:09:37.541162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.540208    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1513277-9bfe-4bfb-9a91-fb3b9ab39ee2" containerName="istio-proxy"
Oct 30 19:09:37.541162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.540225    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2c9ad2-ed90-4a65-adcf-3b2540f128ed" containerName="istio-proxy"
Oct 30 19:09:37.541162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.540238    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ebfc68-3553-4bdb-9393-bc3b582c0bc0" containerName="istio-proxy"
Oct 30 19:09:37.541162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.540249    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62c6f22-f5b9-4061-9a75-ea765cf1a199" containerName="istio-proxy"
Oct 30 19:09:37.548446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.546593    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"
Oct 30 19:09:37.611430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.611383    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"]
Oct 30 19:09:37.636183 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.634983    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vf8l\" (UniqueName: \"kubernetes.io/projected/0aaab9a7-bb44-4ab0-813d-03be5fa24219-kube-api-access-7vf8l\") pod \"payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl\" (UID: \"0aaab9a7-bb44-4ab0-813d-03be5fa24219\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"
Oct 30 19:09:37.636183 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.635174    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aaab9a7-bb44-4ab0-813d-03be5fa24219-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl\" (UID: \"0aaab9a7-bb44-4ab0-813d-03be5fa24219\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"
Oct 30 19:09:37.640165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.636877    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2e004a\""
Oct 30 19:09:37.640165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.636971    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2e004a-dockercfg-6n4nt\""
Oct 30 19:09:37.761202 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.755760    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"]
Oct 30 19:09:37.762682 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.761936    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"
Oct 30 19:09:37.794846 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.779899    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf8l\" (UniqueName: \"kubernetes.io/projected/0aaab9a7-bb44-4ab0-813d-03be5fa24219-kube-api-access-7vf8l\") pod \"payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl\" (UID: \"0aaab9a7-bb44-4ab0-813d-03be5fa24219\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"
Oct 30 19:09:37.794846 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.784326    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aaab9a7-bb44-4ab0-813d-03be5fa24219-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl\" (UID: \"0aaab9a7-bb44-4ab0-813d-03be5fa24219\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"
Oct 30 19:09:37.796986 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.796121    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aaab9a7-bb44-4ab0-813d-03be5fa24219-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl\" (UID: \"0aaab9a7-bb44-4ab0-813d-03be5fa24219\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"
Oct 30 19:09:37.878850 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.871892    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vf8l\" (UniqueName: \"kubernetes.io/projected/0aaab9a7-bb44-4ab0-813d-03be5fa24219-kube-api-access-7vf8l\") pod \"payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl\" (UID: \"0aaab9a7-bb44-4ab0-813d-03be5fa24219\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"
Oct 30 19:09:37.942395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.942312    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"
Oct 30 19:09:37.942894 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.942864    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq29b\" (UniqueName: \"kubernetes.io/projected/af524bf1-8068-45ed-bb37-832eaea6a99a-kube-api-access-cq29b\") pod \"payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s\" (UID: \"af524bf1-8068-45ed-bb37-832eaea6a99a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"
Oct 30 19:09:37.943452 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.943428    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af524bf1-8068-45ed-bb37-832eaea6a99a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s\" (UID: \"af524bf1-8068-45ed-bb37-832eaea6a99a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"
Oct 30 19:09:37.981195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:37.979773    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"]
Oct 30 19:09:38.068604 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:38.068562    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq29b\" (UniqueName: \"kubernetes.io/projected/af524bf1-8068-45ed-bb37-832eaea6a99a-kube-api-access-cq29b\") pod \"payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s\" (UID: \"af524bf1-8068-45ed-bb37-832eaea6a99a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"
Oct 30 19:09:38.069191 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:38.069162    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af524bf1-8068-45ed-bb37-832eaea6a99a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s\" (UID: \"af524bf1-8068-45ed-bb37-832eaea6a99a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"
Oct 30 19:09:38.070323 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:38.070299    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af524bf1-8068-45ed-bb37-832eaea6a99a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s\" (UID: \"af524bf1-8068-45ed-bb37-832eaea6a99a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"
Oct 30 19:09:38.176775 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:38.176741    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq29b\" (UniqueName: \"kubernetes.io/projected/af524bf1-8068-45ed-bb37-832eaea6a99a-kube-api-access-cq29b\") pod \"payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s\" (UID: \"af524bf1-8068-45ed-bb37-832eaea6a99a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"
Oct 30 19:09:38.400938 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:38.400885    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"
Oct 30 19:09:38.617770 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:38.603715    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"]
Oct 30 19:09:38.837317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:38.837177    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"]
Oct 30 19:09:38.842348 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:09:38.842313    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf524bf1_8068_45ed_bb37_832eaea6a99a.slice/crio-c3de788fb4b76bf3deade363f806bb4c749c54930797d6db65090f256f7bd8a4 WatchSource:0}: Error finding container c3de788fb4b76bf3deade363f806bb4c749c54930797d6db65090f256f7bd8a4: Status 404 returned error can't find the container with id c3de788fb4b76bf3deade363f806bb4c749c54930797d6db65090f256f7bd8a4
Oct 30 19:09:39.393548 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:39.393448    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s" event={"ID":"af524bf1-8068-45ed-bb37-832eaea6a99a","Type":"ContainerStarted","Data":"fbdf2ed25b57374782f52790755395435905667cb694d8be1a4eaa1133649bf0"}
Oct 30 19:09:39.393548 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:39.393521    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s" event={"ID":"af524bf1-8068-45ed-bb37-832eaea6a99a","Type":"ContainerStarted","Data":"c3de788fb4b76bf3deade363f806bb4c749c54930797d6db65090f256f7bd8a4"}
Oct 30 19:09:39.394161 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:39.394124    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"
Oct 30 19:09:39.396437 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:39.396407    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl" event={"ID":"0aaab9a7-bb44-4ab0-813d-03be5fa24219","Type":"ContainerStarted","Data":"b1962d2ab216fefea938f50ac798e0910b4f730eea3ecf700f5a7efc41bab373"}
Oct 30 19:09:39.396545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:39.396454    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl" event={"ID":"0aaab9a7-bb44-4ab0-813d-03be5fa24219","Type":"ContainerStarted","Data":"fecdc0f214556e4443bcdcc70b82c7975beb6ccc1a387bb22afb6c933b09b858"}
Oct 30 19:09:39.397262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:39.397239    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"
Oct 30 19:09:39.516030 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:39.515971    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s" podStartSLOduration=2.51595558 podStartE2EDuration="2.51595558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:39.513984702 +0000 UTC m=+1818.157152317" watchObservedRunningTime="2026-10-30 19:09:39.51595558 +0000 UTC m=+1818.159123230"
Oct 30 19:09:39.632591 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:39.632367    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl" podStartSLOduration=2.632348311 podStartE2EDuration="2.632348311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:39.629769761 +0000 UTC m=+1818.272937375" watchObservedRunningTime="2026-10-30 19:09:39.632348311 +0000 UTC m=+1818.275515929"
Oct 30 19:09:47.052008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:47.051970    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"]
Oct 30 19:09:47.101335 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:47.101299    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2be7d8a7-4d5rr"]
Oct 30 19:09:47.478570 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:47.478534    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd62bd5a-0559-41b4-b504-901d6b32782f" path="/var/lib/kubelet/pods/dd62bd5a-0559-41b4-b504-901d6b32782f/volumes"
Oct 30 19:09:50.407340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:50.407310    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"
Oct 30 19:09:51.415118 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:09:51.415084    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"
Oct 30 19:09:53.258827 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:09:53.258794    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:10:01.325715 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.325678    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"]
Oct 30 19:10:01.332312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.332285    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:01.387566 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.387527    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"]
Oct 30 19:10:01.432903 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.432867    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f26b1d-serving-cert\""
Oct 30 19:10:01.433865 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.433839    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-openshift-service-ca\") pod \"maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:01.434061 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.434040    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-metrics-certs\") pod \"maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:01.434247 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.434226    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lthfv\" (UniqueName: \"kubernetes.io/projected/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-kube-api-access-lthfv\") pod \"maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:01.434345 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.434330    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-maas-api-tls\") pod \"maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:01.536195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.536112    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-metrics-certs\") pod \"maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:01.536384 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.536349    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lthfv\" (UniqueName: \"kubernetes.io/projected/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-kube-api-access-lthfv\") pod \"maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:01.536489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.536470    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-maas-api-tls\") pod \"maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:01.536572 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.536554    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-openshift-service-ca\") pod \"maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:01.536695 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:10:01.536673    6014 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-serving-cert: secret "maas-api-e2e-api-a-f26b1d-serving-cert" not found
Oct 30 19:10:01.536769 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:10:01.536740    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-maas-api-tls podName:f52ebf7a-2923-4b71-9989-0f9029fc6dfc nodeName:}" failed. No retries permitted until 2026-10-30 19:10:02.036721581 +0000 UTC m=+1840.679889174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-maas-api-tls") pod "maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq" (UID: "f52ebf7a-2923-4b71-9989-0f9029fc6dfc") : secret "maas-api-e2e-api-a-f26b1d-serving-cert" not found
Oct 30 19:10:01.537277 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.537253    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-openshift-service-ca\") pod \"maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:01.538945 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.538923    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-metrics-certs\") pod \"maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:01.599832 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:01.599183    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lthfv\" (UniqueName: \"kubernetes.io/projected/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-kube-api-access-lthfv\") pod \"maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:02.046176 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:02.046058    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-maas-api-tls\") pod \"maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:02.049021 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:02.048995    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-maas-api-tls\") pod \"maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:02.248834 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:02.248792    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:02.634878 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:02.634848    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"]
Oct 30 19:10:02.635417 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:10:02.635377    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52ebf7a_2923_4b71_9989_0f9029fc6dfc.slice/crio-256e9e5d844dfb09b2fcdddf4035b1a3b93d4b51f3d19de5a0952effa4d67b54 WatchSource:0}: Error finding container 256e9e5d844dfb09b2fcdddf4035b1a3b93d4b51f3d19de5a0952effa4d67b54: Status 404 returned error can't find the container with id 256e9e5d844dfb09b2fcdddf4035b1a3b93d4b51f3d19de5a0952effa4d67b54
Oct 30 19:10:03.624003 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:03.623179    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq" event={"ID":"f52ebf7a-2923-4b71-9989-0f9029fc6dfc","Type":"ContainerStarted","Data":"7c998dbedde824ee2accb29dbdbcc0ef257546795e9bc425f50c19d1c838bf4c"}
Oct 30 19:10:03.624003 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:03.623243    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq" event={"ID":"f52ebf7a-2923-4b71-9989-0f9029fc6dfc","Type":"ContainerStarted","Data":"256e9e5d844dfb09b2fcdddf4035b1a3b93d4b51f3d19de5a0952effa4d67b54"}
Oct 30 19:10:03.624003 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:03.623467    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:03.772181 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:03.772046    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq" podStartSLOduration=1.892442302 podStartE2EDuration="2.77202186s" totalImagesPullingTime="879.579558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:03.766764618 +0000 UTC m=+1842.409932233" watchObservedRunningTime="2026-10-30 19:10:03.77202186 +0000 UTC m=+1842.415189474"
Oct 30 19:10:09.639302 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:09.639273    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:10:11.298400 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.298358    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"]
Oct 30 19:10:11.302900 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.302881    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:11.406382 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.406344    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0b8d0f-serving-cert\""
Oct 30 19:10:11.407090 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.407062    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-openshift-service-ca\") pod \"maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:11.407247 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.407100    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-metrics-certs\") pod \"maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:11.407329 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.407289    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skl82\" (UniqueName: \"kubernetes.io/projected/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-kube-api-access-skl82\") pod \"maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:11.407525 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.407504    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-maas-api-tls\") pod \"maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:11.410366 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.410345    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"]
Oct 30 19:10:11.509561 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.509526    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-maas-api-tls\") pod \"maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:11.509770 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.509661    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-openshift-service-ca\") pod \"maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:11.509770 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:10:11.509681    6014 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-serving-cert: secret "maas-api-e2e-api-a-0b8d0f-serving-cert" not found
Oct 30 19:10:11.509770 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.509693    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-metrics-certs\") pod \"maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:11.509770 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:10:11.509763    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-maas-api-tls podName:55f0fbaa-51ec-407a-bf9d-d14a2bc93156 nodeName:}" failed. No retries permitted until 2026-10-30 19:10:12.009742475 +0000 UTC m=+1850.652910068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-maas-api-tls") pod "maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr" (UID: "55f0fbaa-51ec-407a-bf9d-d14a2bc93156") : secret "maas-api-e2e-api-a-0b8d0f-serving-cert" not found
Oct 30 19:10:11.510003 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.509824    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skl82\" (UniqueName: \"kubernetes.io/projected/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-kube-api-access-skl82\") pod \"maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:11.510529 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.510507    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-openshift-service-ca\") pod \"maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:11.512523 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.512506    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-metrics-certs\") pod \"maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:11.573454 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:11.573425    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skl82\" (UniqueName: \"kubernetes.io/projected/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-kube-api-access-skl82\") pod \"maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:12.017903 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:12.017822    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-maas-api-tls\") pod \"maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:12.020536 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:12.020507    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-maas-api-tls\") pod \"maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:12.217886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:12.217845    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:12.604552 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:12.604488    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"]
Oct 30 19:10:12.605361 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:10:12.605324    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f0fbaa_51ec_407a_bf9d_d14a2bc93156.slice/crio-f943dbb9d533431edc48ce309376e2d8394d273f93dad7144ff678ca271448cd WatchSource:0}: Error finding container f943dbb9d533431edc48ce309376e2d8394d273f93dad7144ff678ca271448cd: Status 404 returned error can't find the container with id f943dbb9d533431edc48ce309376e2d8394d273f93dad7144ff678ca271448cd
Oct 30 19:10:12.708049 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:12.708014    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr" event={"ID":"55f0fbaa-51ec-407a-bf9d-d14a2bc93156","Type":"ContainerStarted","Data":"f943dbb9d533431edc48ce309376e2d8394d273f93dad7144ff678ca271448cd"}
Oct 30 19:10:13.719053 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:13.719014    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr" event={"ID":"55f0fbaa-51ec-407a-bf9d-d14a2bc93156","Type":"ContainerStarted","Data":"2107b03b51d4d7ce3b761a7c20ea3bc0ca7172d63294fd15a9cab7d0e1c35eb9"}
Oct 30 19:10:13.719524 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:13.719189    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:13.839330 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:13.839276    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr" podStartSLOduration=1.965083618 podStartE2EDuration="2.839259064s" totalImagesPullingTime="874.175446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:13.836578096 +0000 UTC m=+1852.479745710" watchObservedRunningTime="2026-10-30 19:10:13.839259064 +0000 UTC m=+1852.482426677"
Oct 30 19:10:15.786228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:15.786192    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"]
Oct 30 19:10:15.786639 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:15.786441    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg" podUID="c954dfe7-18f9-40a9-a921-7c40f023ec98" containerName="maas-api" containerID="cri-o://e790c22f0c9799be4f908dad261786a6244ff89aa5db80f2759adaf8734c54b8" gracePeriod=30
Oct 30 19:10:16.124656 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.124628    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:10:16.191209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.180936    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c954dfe7-18f9-40a9-a921-7c40f023ec98-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c954dfe7-18f9-40a9-a921-7c40f023ec98-openshift-service-ca\") pod \"c954dfe7-18f9-40a9-a921-7c40f023ec98\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") "
Oct 30 19:10:16.191209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.181122    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-metrics-certs\") pod \"c954dfe7-18f9-40a9-a921-7c40f023ec98\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") "
Oct 30 19:10:16.191209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.181206    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c954dfe7-18f9-40a9-a921-7c40f023ec98-kube-api-access-zspqr\" (UniqueName: \"kubernetes.io/projected/c954dfe7-18f9-40a9-a921-7c40f023ec98-kube-api-access-zspqr\") pod \"c954dfe7-18f9-40a9-a921-7c40f023ec98\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") "
Oct 30 19:10:16.191209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.181274    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-maas-api-tls\") pod \"c954dfe7-18f9-40a9-a921-7c40f023ec98\" (UID: \"c954dfe7-18f9-40a9-a921-7c40f023ec98\") "
Oct 30 19:10:16.192784 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.192748    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c954dfe7-18f9-40a9-a921-7c40f023ec98-openshift-service-ca" pod "c954dfe7-18f9-40a9-a921-7c40f023ec98" (UID: "c954dfe7-18f9-40a9-a921-7c40f023ec98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:16.195241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.195214    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c954dfe7-18f9-40a9-a921-7c40f023ec98-kube-api-access-zspqr" pod "c954dfe7-18f9-40a9-a921-7c40f023ec98" (UID: "c954dfe7-18f9-40a9-a921-7c40f023ec98"). InnerVolumeSpecName "kube-api-access-zspqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:16.209312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.209265    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-metrics-certs" pod "c954dfe7-18f9-40a9-a921-7c40f023ec98" (UID: "c954dfe7-18f9-40a9-a921-7c40f023ec98"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:10:16.213659 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.213601    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-maas-api-tls" pod "c954dfe7-18f9-40a9-a921-7c40f023ec98" (UID: "c954dfe7-18f9-40a9-a921-7c40f023ec98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:10:16.283955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.283918    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:16.283955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.283948    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c954dfe7-18f9-40a9-a921-7c40f023ec98-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:16.283955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.283957    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c954dfe7-18f9-40a9-a921-7c40f023ec98-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:16.283955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.283965    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zspqr\" (UniqueName: \"kubernetes.io/projected/c954dfe7-18f9-40a9-a921-7c40f023ec98-kube-api-access-zspqr\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:16.905125 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.905080    6014 generic.go:356] "Generic (PLEG): container finished" podID="c954dfe7-18f9-40a9-a921-7c40f023ec98" containerID="e790c22f0c9799be4f908dad261786a6244ff89aa5db80f2759adaf8734c54b8" exitCode=0
Oct 30 19:10:16.906004 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.905976    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg" event={"ID":"c954dfe7-18f9-40a9-a921-7c40f023ec98","Type":"ContainerDied","Data":"e790c22f0c9799be4f908dad261786a6244ff89aa5db80f2759adaf8734c54b8"}
Oct 30 19:10:16.906240 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.906220    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg" event={"ID":"c954dfe7-18f9-40a9-a921-7c40f023ec98","Type":"ContainerDied","Data":"d18dd57760fa69317e0f05ca3d45bcc50277e404d7351d3dfe0f5ce6e1f10fdb"}
Oct 30 19:10:16.906427 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.906373    6014 scope.go:122] "RemoveContainer" containerID="e790c22f0c9799be4f908dad261786a6244ff89aa5db80f2759adaf8734c54b8"
Oct 30 19:10:16.906755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.906738    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"
Oct 30 19:10:16.930913 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.930870    6014 scope.go:122] "RemoveContainer" containerID="e790c22f0c9799be4f908dad261786a6244ff89aa5db80f2759adaf8734c54b8"
Oct 30 19:10:16.931436 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:10:16.931405    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e790c22f0c9799be4f908dad261786a6244ff89aa5db80f2759adaf8734c54b8\": container with ID starting with e790c22f0c9799be4f908dad261786a6244ff89aa5db80f2759adaf8734c54b8 not found: ID does not exist" containerID="e790c22f0c9799be4f908dad261786a6244ff89aa5db80f2759adaf8734c54b8"
Oct 30 19:10:16.931533 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:16.931447    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e790c22f0c9799be4f908dad261786a6244ff89aa5db80f2759adaf8734c54b8"} err="failed to get container status \"e790c22f0c9799be4f908dad261786a6244ff89aa5db80f2759adaf8734c54b8\": rpc error: code = NotFound desc = could not find container \"e790c22f0c9799be4f908dad261786a6244ff89aa5db80f2759adaf8734c54b8\": container with ID starting with e790c22f0c9799be4f908dad261786a6244ff89aa5db80f2759adaf8734c54b8 not found: ID does not exist"
Oct 30 19:10:17.079900 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:17.079867    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"]
Oct 30 19:10:17.128814 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:17.128782    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4501c8d4-84cc94c994-n4sbg"]
Oct 30 19:10:17.480010 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:17.479978    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c954dfe7-18f9-40a9-a921-7c40f023ec98" path="/var/lib/kubelet/pods/c954dfe7-18f9-40a9-a921-7c40f023ec98/volumes"
Oct 30 19:10:19.743840 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:19.743812    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:10:23.260910 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:10:23.260870    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:10:24.130324 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:24.130287    6014 scope.go:122] "RemoveContainer" containerID="7b912b22e2d19dc05a7d4857620e9616fbc05a64b20181a85482b70e4e7225f0"
Oct 30 19:10:24.140927 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:24.140898    6014 scope.go:122] "RemoveContainer" containerID="13a46748040eb62f7d3b248f10867c65d0a46ae144f1f838077679aa272dbaa7"
Oct 30 19:10:24.151304 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:24.151283    6014 scope.go:122] "RemoveContainer" containerID="578d9a07b4958e33045900f03913d935967681b157323f709cc8a39dac59a6b7"
Oct 30 19:10:26.326174 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.326102    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"]
Oct 30 19:10:26.326914 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.326899    6014 cpu_manager.go:414] "containerMap: removing container" podUID="c954dfe7-18f9-40a9-a921-7c40f023ec98" containerName="maas-api"
Oct 30 19:10:26.326971 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.326915    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c954dfe7-18f9-40a9-a921-7c40f023ec98" containerName="maas-api"
Oct 30 19:10:26.327043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.327033    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="c954dfe7-18f9-40a9-a921-7c40f023ec98" containerName="maas-api"
Oct 30 19:10:26.333394 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.333376    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.447167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.386555    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-322e97-openshift-default-dockercfg-ccnnj\""
Oct 30 19:10:26.447167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.404261    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-podinfo\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.447167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.404344    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-credential-socket\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.447167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.404389    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-envoy\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.447167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.404428    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-certs\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.447167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.404474    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-token\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.447167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.404561    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9fsq\" (UniqueName: \"kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-kube-api-access-v9fsq\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.447167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.404689    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd41d427-10c0-4e58-93fb-87d7d4296904-istiod-ca-cert\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.447167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.404816    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-data\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.447167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.404867    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-socket\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.522229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.521434    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-data\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.522229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.521511    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-socket\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.522229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.521586    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-podinfo\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.522229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.521655    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-credential-socket\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.522229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.521701    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-envoy\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.522229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.521741    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-certs\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.522229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.521790    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-token\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.522229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.521883    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9fsq\" (UniqueName: \"kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-kube-api-access-v9fsq\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.522229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.522021    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd41d427-10c0-4e58-93fb-87d7d4296904-istiod-ca-cert\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.536584 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.536533    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-data\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.538672 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.538645    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-socket\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.539364 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.539335    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-certs\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.540063 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.540033    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-credential-socket\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.557211 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.545713    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd41d427-10c0-4e58-93fb-87d7d4296904-istiod-ca-cert\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.557441 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.546881    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-podinfo\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.562734 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.549876    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"]
Oct 30 19:10:26.562958 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.559790    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-envoy\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.598311 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.598232    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-token\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.603128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.603055    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9fsq\" (UniqueName: \"kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-kube-api-access-v9fsq\") pod \"e2e-inf-a-322e97-openshift-default-856dffd7-kb7db\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") " pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:26.661984 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:26.661950    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:27.126555 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:27.126524    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"]
Oct 30 19:10:27.127541 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:10:27.127518    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd41d427_10c0_4e58_93fb_87d7d4296904.slice/crio-ce76965677dab09b8b36d16a4257ff95448266a29c4d04c2db8d4c1921992569 WatchSource:0}: Error finding container ce76965677dab09b8b36d16a4257ff95448266a29c4d04c2db8d4c1921992569: Status 404 returned error can't find the container with id ce76965677dab09b8b36d16a4257ff95448266a29c4d04c2db8d4c1921992569
Oct 30 19:10:27.130559 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:27.130526    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:10:27.130623 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:27.130593    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:10:27.130659 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:27.130623    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:10:27.396891 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:27.396804    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:10:28.056208 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:28.051728    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" event={"ID":"bd41d427-10c0-4e58-93fb-87d7d4296904","Type":"ContainerStarted","Data":"5b459a0bbde9234158a33ade757e6a107e957a6c1ae4ff7cbb9c94859548a498"}
Oct 30 19:10:28.056208 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:28.051798    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" event={"ID":"bd41d427-10c0-4e58-93fb-87d7d4296904","Type":"ContainerStarted","Data":"ce76965677dab09b8b36d16a4257ff95448266a29c4d04c2db8d4c1921992569"}
Oct 30 19:10:28.228106 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:28.228039    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" podStartSLOduration=2.228017729 podStartE2EDuration="2.228017729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:28.224539459 +0000 UTC m=+1866.867707074" watchObservedRunningTime="2026-10-30 19:10:28.228017729 +0000 UTC m=+1866.871185344"
Oct 30 19:10:28.663112 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:28.663075    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:28.669817 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:28.669789    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:29.076122 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:29.076087    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:29.077242 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:29.077221    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:10:38.065416 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:38.065382    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"]
Oct 30 19:10:38.114015 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:38.113980    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f170e48-s5p2v"]
Oct 30 19:10:39.252040 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:39.252007    6014 generic.go:356] "Generic (PLEG): container finished" podID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerID="61437ab78b927ffa560c1d02fc53c5f1fd9f3b0e6a4e804208b27d7b89eb9c39" exitCode=0
Oct 30 19:10:39.252504 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:39.252078    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" event={"ID":"69d8cf2e-7a19-49f8-8967-2deeff9ed05c","Type":"ContainerDied","Data":"61437ab78b927ffa560c1d02fc53c5f1fd9f3b0e6a4e804208b27d7b89eb9c39"}
Oct 30 19:10:39.252611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:39.252594    6014 scope.go:122] "RemoveContainer" containerID="61437ab78b927ffa560c1d02fc53c5f1fd9f3b0e6a4e804208b27d7b89eb9c39"
Oct 30 19:10:39.254457 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:39.254425    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:10:39.254574 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:39.254510    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:10:39.254574 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:39.254555    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:10:39.482985 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:39.482949    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c6362f-0454-42f1-87fd-492422802d4a" path="/var/lib/kubelet/pods/b1c6362f-0454-42f1-87fd-492422802d4a/volumes"
Oct 30 19:10:40.281208 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:40.281165    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" event={"ID":"69d8cf2e-7a19-49f8-8967-2deeff9ed05c","Type":"ContainerStarted","Data":"a6a88c8725b79fa1ad933e1ea48477321d71bdfe3b5a7ade02de5f44e3adf157"}
Oct 30 19:10:40.383640 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:40.383612    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:10:40.572226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:40.565552    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:10:40.572226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:40.571267    6014 patch_prober.go:29] interesting pod/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:10:40.572226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:40.571330    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 19:10:41.569169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:41.565565    6014 patch_prober.go:29] interesting pod/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:10:41.569169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:41.565641    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 19:10:42.552598 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:42.552566    6014 patch_prober.go:29] interesting pod/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:10:42.552895 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:42.552849    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 19:10:43.567166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:43.566776    6014 patch_prober.go:29] interesting pod/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:10:43.567166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:43.566877    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 19:10:44.569177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:44.568437    6014 patch_prober.go:29] interesting pod/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:10:44.569177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:44.568516    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 19:10:45.550189 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:45.550038    6014 patch_prober.go:29] interesting pod/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:10:45.550189 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:45.550108    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 19:10:46.554638 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:46.554605    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:10:46.555888 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:46.555870    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:10:53.260466 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:10:53.260430    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:10:53.467474 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:53.467437    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"]
Oct 30 19:10:53.467663 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:53.467647    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8" podUID="311a8fef-9742-4418-b638-ee0f8a474764" containerName="istio-proxy" containerID="cri-o://9371167ab8afbed92a3f023e8d5d5bcc8e8c4a3480dec6a40aee528f1df5b1c6" gracePeriod=30
Oct 30 19:10:57.630857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:57.630821    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:10:57.631269 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:57.630883    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8" podUID="311a8fef-9742-4418-b638-ee0f8a474764" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:10:58.762862 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.762831    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:10:58.826767 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.826737    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-data\") pod \"311a8fef-9742-4418-b638-ee0f8a474764\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") "
Oct 30 19:10:58.826933 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.826779    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/311a8fef-9742-4418-b638-ee0f8a474764-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311a8fef-9742-4418-b638-ee0f8a474764-istio-podinfo\") pod \"311a8fef-9742-4418-b638-ee0f8a474764\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") "
Oct 30 19:10:58.826933 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.826842    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311a8fef-9742-4418-b638-ee0f8a474764-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311a8fef-9742-4418-b638-ee0f8a474764-istiod-ca-cert\") pod \"311a8fef-9742-4418-b638-ee0f8a474764\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") "
Oct 30 19:10:58.826933 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.826859    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-kube-api-access-rm9gd\" (UniqueName: \"kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-kube-api-access-rm9gd\") pod \"311a8fef-9742-4418-b638-ee0f8a474764\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") "
Oct 30 19:10:58.827088 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.827057    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-data" pod "311a8fef-9742-4418-b638-ee0f8a474764" (UID: "311a8fef-9742-4418-b638-ee0f8a474764"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:58.827242 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.827222    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-socket\") pod \"311a8fef-9742-4418-b638-ee0f8a474764\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") "
Oct 30 19:10:58.827322 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.827262    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-envoy\") pod \"311a8fef-9742-4418-b638-ee0f8a474764\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") "
Oct 30 19:10:58.827322 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.827316    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-istio-token\" (UniqueName: \"kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-istio-token\") pod \"311a8fef-9742-4418-b638-ee0f8a474764\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") "
Oct 30 19:10:58.827453 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.827335    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311a8fef-9742-4418-b638-ee0f8a474764-istiod-ca-cert" pod "311a8fef-9742-4418-b638-ee0f8a474764" (UID: "311a8fef-9742-4418-b638-ee0f8a474764"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:58.827453 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.827369    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-certs\") pod \"311a8fef-9742-4418-b638-ee0f8a474764\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") "
Oct 30 19:10:58.827453 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.827422    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-credential-socket\") pod \"311a8fef-9742-4418-b638-ee0f8a474764\" (UID: \"311a8fef-9742-4418-b638-ee0f8a474764\") "
Oct 30 19:10:58.827615 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.827507    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-socket" pod "311a8fef-9742-4418-b638-ee0f8a474764" (UID: "311a8fef-9742-4418-b638-ee0f8a474764"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:58.827786 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.827749    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-certs" pod "311a8fef-9742-4418-b638-ee0f8a474764" (UID: "311a8fef-9742-4418-b638-ee0f8a474764"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:58.827957 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.827929    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-credential-socket" pod "311a8fef-9742-4418-b638-ee0f8a474764" (UID: "311a8fef-9742-4418-b638-ee0f8a474764"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:58.828344 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.828322    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311a8fef-9742-4418-b638-ee0f8a474764-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:58.828408 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.828352    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:58.828408 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.828367    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:58.828408 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.828382    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:58.828408 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.828394    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:58.829451 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.829431    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-kube-api-access-rm9gd" pod "311a8fef-9742-4418-b638-ee0f8a474764" (UID: "311a8fef-9742-4418-b638-ee0f8a474764"). InnerVolumeSpecName "kube-api-access-rm9gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:58.829779 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.829754    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-istio-token" pod "311a8fef-9742-4418-b638-ee0f8a474764" (UID: "311a8fef-9742-4418-b638-ee0f8a474764"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:58.829903 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.829883    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-envoy" pod "311a8fef-9742-4418-b638-ee0f8a474764" (UID: "311a8fef-9742-4418-b638-ee0f8a474764"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:58.829903 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.829894    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/311a8fef-9742-4418-b638-ee0f8a474764-istio-podinfo" pod "311a8fef-9742-4418-b638-ee0f8a474764" (UID: "311a8fef-9742-4418-b638-ee0f8a474764"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:10:58.929857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.929824    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311a8fef-9742-4418-b638-ee0f8a474764-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:58.929857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.929852    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:58.929857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.929863    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311a8fef-9742-4418-b638-ee0f8a474764-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:58.930106 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:58.929874    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm9gd\" (UniqueName: \"kubernetes.io/projected/311a8fef-9742-4418-b638-ee0f8a474764-kube-api-access-rm9gd\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:59.444823 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.444789    6014 generic.go:356] "Generic (PLEG): container finished" podID="311a8fef-9742-4418-b638-ee0f8a474764" containerID="9371167ab8afbed92a3f023e8d5d5bcc8e8c4a3480dec6a40aee528f1df5b1c6" exitCode=0
Oct 30 19:10:59.445042 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.444842    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"
Oct 30 19:10:59.445042 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.444875    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8" event={"ID":"311a8fef-9742-4418-b638-ee0f8a474764","Type":"ContainerDied","Data":"9371167ab8afbed92a3f023e8d5d5bcc8e8c4a3480dec6a40aee528f1df5b1c6"}
Oct 30 19:10:59.445042 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.444971    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8" event={"ID":"311a8fef-9742-4418-b638-ee0f8a474764","Type":"ContainerDied","Data":"18a4c2ee52b2ef4f1363865ec335aaed51c7261f36c1e6d8abd6eaf211f8e3dc"}
Oct 30 19:10:59.445042 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.445001    6014 scope.go:122] "RemoveContainer" containerID="9371167ab8afbed92a3f023e8d5d5bcc8e8c4a3480dec6a40aee528f1df5b1c6"
Oct 30 19:10:59.457418 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.457399    6014 scope.go:122] "RemoveContainer" containerID="9371167ab8afbed92a3f023e8d5d5bcc8e8c4a3480dec6a40aee528f1df5b1c6"
Oct 30 19:10:59.457782 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:10:59.457755    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9371167ab8afbed92a3f023e8d5d5bcc8e8c4a3480dec6a40aee528f1df5b1c6\": container with ID starting with 9371167ab8afbed92a3f023e8d5d5bcc8e8c4a3480dec6a40aee528f1df5b1c6 not found: ID does not exist" containerID="9371167ab8afbed92a3f023e8d5d5bcc8e8c4a3480dec6a40aee528f1df5b1c6"
Oct 30 19:10:59.457908 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.457795    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9371167ab8afbed92a3f023e8d5d5bcc8e8c4a3480dec6a40aee528f1df5b1c6"} err="failed to get container status \"9371167ab8afbed92a3f023e8d5d5bcc8e8c4a3480dec6a40aee528f1df5b1c6\": rpc error: code = NotFound desc = could not find container \"9371167ab8afbed92a3f023e8d5d5bcc8e8c4a3480dec6a40aee528f1df5b1c6\": container with ID starting with 9371167ab8afbed92a3f023e8d5d5bcc8e8c4a3480dec6a40aee528f1df5b1c6 not found: ID does not exist"
Oct 30 19:10:59.613499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.613465    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"]
Oct 30 19:10:59.616946 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.616915    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-w6vz8"]
Oct 30 19:10:59.806617 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.806540    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"]
Oct 30 19:10:59.807360 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.807346    6014 cpu_manager.go:414] "containerMap: removing container" podUID="311a8fef-9742-4418-b638-ee0f8a474764" containerName="istio-proxy"
Oct 30 19:10:59.807426 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.807363    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311a8fef-9742-4418-b638-ee0f8a474764" containerName="istio-proxy"
Oct 30 19:10:59.807513 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.807504    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="311a8fef-9742-4418-b638-ee0f8a474764" containerName="istio-proxy"
Oct 30 19:10:59.812270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.812248    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:10:59.864401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.864368    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2e004a-serving-cert\""
Oct 30 19:10:59.867780 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.867749    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"]
Oct 30 19:10:59.944539 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.944498    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-metrics-certs\") pod \"maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:10:59.944749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.944621    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x4kc\" (UniqueName: \"kubernetes.io/projected/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-kube-api-access-8x4kc\") pod \"maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:10:59.944828 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.944790    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:10:59.944891 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:10:59.944863    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:11:00.046503 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:00.046468    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:11:00.046723 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:00.046528    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:11:00.046723 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:11:00.046670    6014 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-serving-cert: secret "maas-api-e2e-inf-a-2e004a-serving-cert" not found
Oct 30 19:11:00.046847 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:11:00.046745    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-maas-api-tls podName:97e8b1ea-88f1-4f81-80b1-2b7c81e6926e nodeName:}" failed. No retries permitted until 2026-10-30 19:11:00.546726567 +0000 UTC m=+1899.189894180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-maas-api-tls") pod "maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b" (UID: "97e8b1ea-88f1-4f81-80b1-2b7c81e6926e") : secret "maas-api-e2e-inf-a-2e004a-serving-cert" not found
Oct 30 19:11:00.046847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:00.046790    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-metrics-certs\") pod \"maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:11:00.046847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:00.046837    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x4kc\" (UniqueName: \"kubernetes.io/projected/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-kube-api-access-8x4kc\") pod \"maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:11:00.047211 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:00.047192    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:11:00.049393 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:00.049368    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-metrics-certs\") pod \"maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:11:00.106531 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:00.106498    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x4kc\" (UniqueName: \"kubernetes.io/projected/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-kube-api-access-8x4kc\") pod \"maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:11:00.556705 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:00.556573    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:11:00.559160 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:00.559111    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:11:00.727965 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:00.727933    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:11:01.109192 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:01.109167    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"]
Oct 30 19:11:01.111067 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:11:01.111038    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e8b1ea_88f1_4f81_80b1_2b7c81e6926e.slice/crio-ac5a639a269a26f813dc690a6f28782f9feec279a1b398f3290080a6052e0b8a WatchSource:0}: Error finding container ac5a639a269a26f813dc690a6f28782f9feec279a1b398f3290080a6052e0b8a: Status 404 returned error can't find the container with id ac5a639a269a26f813dc690a6f28782f9feec279a1b398f3290080a6052e0b8a
Oct 30 19:11:01.464647 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:01.464557    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b" event={"ID":"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e","Type":"ContainerStarted","Data":"ac5a639a269a26f813dc690a6f28782f9feec279a1b398f3290080a6052e0b8a"}
Oct 30 19:11:01.479711 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:01.479679    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311a8fef-9742-4418-b638-ee0f8a474764" path="/var/lib/kubelet/pods/311a8fef-9742-4418-b638-ee0f8a474764/volumes"
Oct 30 19:11:02.474653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:02.474612    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b" event={"ID":"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e","Type":"ContainerStarted","Data":"fd1d8f3b98dd3b3e080f48ecec733594c66f6727be5ca881166b10309dc988f0"}
Oct 30 19:11:02.475053 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:02.474691    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:11:02.593330 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:02.593272    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b" podStartSLOduration=2.700880259 podStartE2EDuration="3.593257532s" totalImagesPullingTime="892.377273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:02.588989986 +0000 UTC m=+1901.232157600" watchObservedRunningTime="2026-10-30 19:11:02.593257532 +0000 UTC m=+1901.236425143"
Oct 30 19:11:08.522165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:08.521332    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:11:08.917286 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:08.917249    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:11:20.678428 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:20.678334    6014 generic.go:356] "Generic (PLEG): container finished" podID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerID="2e7bbe2992568a048be8347dfa6ed2188d3b0ee8b67ef2d45d6f28630e0ef7a2" exitCode=0
Oct 30 19:11:20.678428 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:20.678400    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" event={"ID":"07a2f9fa-4621-45b8-84a8-2aa9721df791","Type":"ContainerDied","Data":"2e7bbe2992568a048be8347dfa6ed2188d3b0ee8b67ef2d45d6f28630e0ef7a2"}
Oct 30 19:11:20.678955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:20.678939    6014 scope.go:122] "RemoveContainer" containerID="2e7bbe2992568a048be8347dfa6ed2188d3b0ee8b67ef2d45d6f28630e0ef7a2"
Oct 30 19:11:20.681025 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:20.680998    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:20.681114 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:20.681075    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:20.681114 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:20.681109    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:21.699052 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:21.699008    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" event={"ID":"07a2f9fa-4621-45b8-84a8-2aa9721df791","Type":"ContainerStarted","Data":"c3637d43263f335f5006257860d004e8e4137bcb26b654296e8123ca0b0766ab"}
Oct 30 19:11:22.642329 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:22.642296    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:11:22.642533 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:22.642427    6014 patch_prober.go:29] interesting pod/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:11:22.642533 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:22.642482    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 19:11:23.284245 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:11:23.284209    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:11:23.642458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:23.642427    6014 patch_prober.go:29] interesting pod/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:11:23.642628 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:23.642479    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 19:11:24.219156 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:24.219114    6014 scope.go:122] "RemoveContainer" containerID="f4fbc7657f0fe20debea765c6321f6d120cd471efab544ec06ea51df7204631e"
Oct 30 19:11:24.230257 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:24.230234    6014 scope.go:122] "RemoveContainer" containerID="65c903b1dcb32bde1d63e916294fa2a5ec66f2d9e4f519f0b265ec57569970e0"
Oct 30 19:11:24.642855 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:24.642822    6014 patch_prober.go:29] interesting pod/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:11:24.643282 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:24.642877    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 19:11:25.642786 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:25.642753    6014 patch_prober.go:29] interesting pod/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:11:25.642976 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:25.642805    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 19:11:26.642942 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:26.642909    6014 patch_prober.go:29] interesting pod/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:11:26.643120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:26.642963    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 19:11:27.647785 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:27.647749    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:11:27.648253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:27.648235    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:11:27.649099 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:27.649077    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:11:36.767170 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:36.766616    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"]
Oct 30 19:11:36.796169 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:36.793998    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:36.954172 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:36.934577    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-903a82-openshift-default-dockercfg-t4bdf\""
Oct 30 19:11:37.021251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:36.997343    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istiod-ca-cert\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.021251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:36.997409    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-envoy\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.021251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:36.997493    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-token\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.021251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:36.997530    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xddjn\" (UniqueName: \"kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-kube-api-access-xddjn\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.021251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:36.997575    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-socket\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.021251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:36.997611    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-certs\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.021251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:36.997694    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-credential-socket\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.021251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:36.997803    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-podinfo\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.021251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:36.997883    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-data\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.040952 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.040908    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"]
Oct 30 19:11:37.099537 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.099489    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istiod-ca-cert\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.099721 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.099574    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-envoy\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.099721 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.099676    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-token\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.099721 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.099719    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xddjn\" (UniqueName: \"kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-kube-api-access-xddjn\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.099912 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.099766    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-socket\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.099912 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.099805    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-certs\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.099912 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.099877    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-credential-socket\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.100084 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.099936    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-podinfo\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.100084 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.100020    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-data\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.100325 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.100292    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-socket\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.100423 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.100399    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istiod-ca-cert\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.100632 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.100606    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-data\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.100723 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.100682    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-certs\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.100951 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.100890    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-credential-socket\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.103420 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.103368    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-envoy\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.103529 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.103444    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-podinfo\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.181719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.181690    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-token\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.182839 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.182814    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xddjn\" (UniqueName: \"kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-kube-api-access-xddjn\") pod \"e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") " pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.430287 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.430099    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:37.925926 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.925888    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"]
Oct 30 19:11:37.978224 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:11:37.978193    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode693b661_2b77_4bf5_82f7_b11fd1e570bb.slice/crio-b020b88c02170cc2dff750a0002252a1b92250c7922f86344057bf5f01ef971b WatchSource:0}: Error finding container b020b88c02170cc2dff750a0002252a1b92250c7922f86344057bf5f01ef971b: Status 404 returned error can't find the container with id b020b88c02170cc2dff750a0002252a1b92250c7922f86344057bf5f01ef971b
Oct 30 19:11:37.981931 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.981880    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:37.982057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.981998    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:37.982057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:37.982043    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:38.783208 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:38.783104    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"]
Oct 30 19:11:38.970165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:38.967133    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h" event={"ID":"e693b661-2b77-4bf5-82f7-b11fd1e570bb","Type":"ContainerStarted","Data":"62696aee7734892c47050c55661aaac6d53cdba08b03020d45c2355cace1640d"}
Oct 30 19:11:38.970165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:38.967228    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h" event={"ID":"e693b661-2b77-4bf5-82f7-b11fd1e570bb","Type":"ContainerStarted","Data":"b020b88c02170cc2dff750a0002252a1b92250c7922f86344057bf5f01ef971b"}
Oct 30 19:11:39.107252 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:39.107192    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h" podStartSLOduration=3.107171949 podStartE2EDuration="3.107171949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:39.089597254 +0000 UTC m=+1937.732764869" watchObservedRunningTime="2026-10-30 19:11:39.107171949 +0000 UTC m=+1937.750339563"
Oct 30 19:11:39.431176 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:39.431052    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:39.438241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:39.438210    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:39.981551 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:39.981519    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:39.982658 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:39.982637    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:11:43.783730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:43.783668    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv" podUID="75587314-c5fa-4ce1-8a13-dced50dde605" containerName="payload-pre-processing" containerID="cri-o://5280ac327f9af4c744cfd9c0cb95a318c4a14288e0a0808672975054779de451" gracePeriod=25
Oct 30 19:11:45.028251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:45.028216    6014 generic.go:356] "Generic (PLEG): container finished" podID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerID="cd629341e9b682c67cb34fdec95952f1a47018a6dd68c251c594f7e6ba365728" exitCode=0
Oct 30 19:11:45.028709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:45.028299    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk" event={"ID":"f081490d-ac76-40cc-9df9-14aafe2726ad","Type":"ContainerDied","Data":"cd629341e9b682c67cb34fdec95952f1a47018a6dd68c251c594f7e6ba365728"}
Oct 30 19:11:45.028920 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:45.028899    6014 scope.go:122] "RemoveContainer" containerID="cd629341e9b682c67cb34fdec95952f1a47018a6dd68c251c594f7e6ba365728"
Oct 30 19:11:45.030862 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:45.030835    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:45.030939 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:45.030903    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:45.030984 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:45.030938    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:46.041465 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:46.041431    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk" event={"ID":"f081490d-ac76-40cc-9df9-14aafe2726ad","Type":"ContainerStarted","Data":"d892fd66ebd609b79c6bd7c5ef6c1d9e1dda24ff5ed58542b798151f24929cc2"}
Oct 30 19:11:46.618358 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:46.618318    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:11:46.624983 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:46.624954    6014 patch_prober.go:29] interesting pod/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:11:46.625164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:46.625019    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk" podUID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 19:11:47.308639 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:47.308601    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:11:47.619075 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:47.619038    6014 patch_prober.go:29] interesting pod/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:11:47.619410 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:47.619376    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk" podUID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 19:11:48.620299 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:48.618625    6014 patch_prober.go:29] interesting pod/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:11:48.620299 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:48.618673    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk" podUID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 19:11:49.618385 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:49.618352    6014 patch_prober.go:29] interesting pod/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:11:49.618617 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:49.618417    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk" podUID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 19:11:50.618269 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:50.618234    6014 patch_prober.go:29] interesting pod/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:11:50.618675 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:50.618304    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk" podUID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 19:11:51.624944 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:51.624908    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:11:51.626200 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:51.626179    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:11:51.950287 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:51.950109    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv" podUID="75587314-c5fa-4ce1-8a13-dced50dde605" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:11:53.259158 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:11:53.259103    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:11:57.446944 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:11:57.446912    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:12:01.949776 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:01.949741    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv" podUID="75587314-c5fa-4ce1-8a13-dced50dde605" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:12:04.084846 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.084815    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"
Oct 30 19:12:04.140430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.140398    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75587314-c5fa-4ce1-8a13-dced50dde605-kube-api-access-8298r\" (UniqueName: \"kubernetes.io/projected/75587314-c5fa-4ce1-8a13-dced50dde605-kube-api-access-8298r\") pod \"75587314-c5fa-4ce1-8a13-dced50dde605\" (UID: \"75587314-c5fa-4ce1-8a13-dced50dde605\") "
Oct 30 19:12:04.140627 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.140488    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75587314-c5fa-4ce1-8a13-dced50dde605-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75587314-c5fa-4ce1-8a13-dced50dde605-plugins-config-volume\") pod \"75587314-c5fa-4ce1-8a13-dced50dde605\" (UID: \"75587314-c5fa-4ce1-8a13-dced50dde605\") "
Oct 30 19:12:04.140873 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.140850    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75587314-c5fa-4ce1-8a13-dced50dde605-plugins-config-volume" pod "75587314-c5fa-4ce1-8a13-dced50dde605" (UID: "75587314-c5fa-4ce1-8a13-dced50dde605"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:04.141235 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.141215    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75587314-c5fa-4ce1-8a13-dced50dde605-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:04.142720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.142700    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75587314-c5fa-4ce1-8a13-dced50dde605-kube-api-access-8298r" pod "75587314-c5fa-4ce1-8a13-dced50dde605" (UID: "75587314-c5fa-4ce1-8a13-dced50dde605"). InnerVolumeSpecName "kube-api-access-8298r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:04.238887 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.238855    6014 generic.go:356] "Generic (PLEG): container finished" podID="75587314-c5fa-4ce1-8a13-dced50dde605" containerID="5280ac327f9af4c744cfd9c0cb95a318c4a14288e0a0808672975054779de451" exitCode=0
Oct 30 19:12:04.239116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.238917    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"
Oct 30 19:12:04.239116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.238941    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv" event={"ID":"75587314-c5fa-4ce1-8a13-dced50dde605","Type":"ContainerDied","Data":"5280ac327f9af4c744cfd9c0cb95a318c4a14288e0a0808672975054779de451"}
Oct 30 19:12:04.239116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.238993    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv" event={"ID":"75587314-c5fa-4ce1-8a13-dced50dde605","Type":"ContainerDied","Data":"b205a664dc8cff4dc12b33085e4f8f3b6bbf6a93bd22e0aa474057d960f8332a"}
Oct 30 19:12:04.239116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.239017    6014 scope.go:122] "RemoveContainer" containerID="5280ac327f9af4c744cfd9c0cb95a318c4a14288e0a0808672975054779de451"
Oct 30 19:12:04.243065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.243036    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8298r\" (UniqueName: \"kubernetes.io/projected/75587314-c5fa-4ce1-8a13-dced50dde605-kube-api-access-8298r\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:04.251961 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.251931    6014 scope.go:122] "RemoveContainer" containerID="5280ac327f9af4c744cfd9c0cb95a318c4a14288e0a0808672975054779de451"
Oct 30 19:12:04.255539 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:12:04.254433    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5280ac327f9af4c744cfd9c0cb95a318c4a14288e0a0808672975054779de451\": container with ID starting with 5280ac327f9af4c744cfd9c0cb95a318c4a14288e0a0808672975054779de451 not found: ID does not exist" containerID="5280ac327f9af4c744cfd9c0cb95a318c4a14288e0a0808672975054779de451"
Oct 30 19:12:04.255539 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.254475    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5280ac327f9af4c744cfd9c0cb95a318c4a14288e0a0808672975054779de451"} err="failed to get container status \"5280ac327f9af4c744cfd9c0cb95a318c4a14288e0a0808672975054779de451\": rpc error: code = NotFound desc = could not find container \"5280ac327f9af4c744cfd9c0cb95a318c4a14288e0a0808672975054779de451\": container with ID starting with 5280ac327f9af4c744cfd9c0cb95a318c4a14288e0a0808672975054779de451 not found: ID does not exist"
Oct 30 19:12:04.404994 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.404956    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"]
Oct 30 19:12:04.409487 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:04.409463    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-qk6tv"]
Oct 30 19:12:05.478566 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:05.478537    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75587314-c5fa-4ce1-8a13-dced50dde605" path="/var/lib/kubelet/pods/75587314-c5fa-4ce1-8a13-dced50dde605/volumes"
Oct 30 19:12:23.051077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:23.050986    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"]
Oct 30 19:12:23.100303 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:23.100269    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7cd214-n46qz"]
Oct 30 19:12:23.260716 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:12:23.260679    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:12:23.479321 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:23.479293    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af035e36-bc85-491a-a20b-b1451412fc15" path="/var/lib/kubelet/pods/af035e36-bc85-491a-a20b-b1451412fc15/volumes"
Oct 30 19:12:24.301185 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:24.301149    6014 scope.go:122] "RemoveContainer" containerID="0254be1caba1bb05bc4c7ac0baf2ac6d2956ba7c1ad7570082589fa3d9723482"
Oct 30 19:12:39.561684 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:39.561652    6014 generic.go:356] "Generic (PLEG): container finished" podID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerID="5b459a0bbde9234158a33ade757e6a107e957a6c1ae4ff7cbb9c94859548a498" exitCode=0
Oct 30 19:12:39.562204 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:39.561740    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" event={"ID":"bd41d427-10c0-4e58-93fb-87d7d4296904","Type":"ContainerDied","Data":"5b459a0bbde9234158a33ade757e6a107e957a6c1ae4ff7cbb9c94859548a498"}
Oct 30 19:12:39.562254 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:39.562212    6014 scope.go:122] "RemoveContainer" containerID="5b459a0bbde9234158a33ade757e6a107e957a6c1ae4ff7cbb9c94859548a498"
Oct 30 19:12:39.564090 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:39.564063    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:12:39.564177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:39.564151    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:12:39.564224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:39.564186    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:12:40.583342 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:40.583303    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" event={"ID":"bd41d427-10c0-4e58-93fb-87d7d4296904","Type":"ContainerStarted","Data":"29ab93bcdc570cc990aae2e2b1c2740d7c72a9406e7bad811e9a5e66584ee2fe"}
Oct 30 19:12:40.669303 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:40.669204    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:12:40.672166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:40.670656    6014 patch_prober.go:29] interesting pod/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:12:40.672166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:40.670729    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 19:12:41.672311 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:41.672270    6014 patch_prober.go:29] interesting pod/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:12:41.672862 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:41.672350    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 19:12:42.670165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:42.669764    6014 patch_prober.go:29] interesting pod/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:12:42.670165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:42.669838    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 19:12:43.668173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:43.668113    6014 patch_prober.go:29] interesting pod/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:12:43.668818 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:43.668781    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 19:12:44.076508 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:44.076470    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:12:44.664578 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:44.664538    6014 patch_prober.go:29] interesting pod/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:12:44.664891 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:44.664859    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 19:12:45.685162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:45.681362    6014 patch_prober.go:29] interesting pod/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:12:45.685162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:45.681441    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 19:12:46.060985 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:46.060875    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"]
Oct 30 19:12:46.168458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:46.168421    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"]
Oct 30 19:12:46.667957 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:46.667926    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:12:46.669272 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:46.669249    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:12:51.064443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.064396    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s" podUID="af524bf1-8068-45ed-bb37-832eaea6a99a" containerName="payload-processing" containerID="cri-o://fbdf2ed25b57374782f52790755395435905667cb694d8be1a4eaa1133649bf0" gracePeriod=25
Oct 30 19:12:51.169750 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.169700    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl" podUID="0aaab9a7-bb44-4ab0-813d-03be5fa24219" containerName="payload-pre-processing" containerID="cri-o://b1962d2ab216fefea938f50ac798e0910b4f730eea3ecf700f5a7efc41bab373" gracePeriod=25
Oct 30 19:12:51.368757 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.368317    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"
Oct 30 19:12:51.406355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.406316    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af524bf1-8068-45ed-bb37-832eaea6a99a-kube-api-access-cq29b\" (UniqueName: \"kubernetes.io/projected/af524bf1-8068-45ed-bb37-832eaea6a99a-kube-api-access-cq29b\") pod \"af524bf1-8068-45ed-bb37-832eaea6a99a\" (UID: \"af524bf1-8068-45ed-bb37-832eaea6a99a\") "
Oct 30 19:12:51.406574 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.406440    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af524bf1-8068-45ed-bb37-832eaea6a99a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af524bf1-8068-45ed-bb37-832eaea6a99a-plugins-config-volume\") pod \"af524bf1-8068-45ed-bb37-832eaea6a99a\" (UID: \"af524bf1-8068-45ed-bb37-832eaea6a99a\") "
Oct 30 19:12:51.407238 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.407204    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af524bf1-8068-45ed-bb37-832eaea6a99a-plugins-config-volume" pod "af524bf1-8068-45ed-bb37-832eaea6a99a" (UID: "af524bf1-8068-45ed-bb37-832eaea6a99a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:51.410664 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.410633    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af524bf1-8068-45ed-bb37-832eaea6a99a-kube-api-access-cq29b" pod "af524bf1-8068-45ed-bb37-832eaea6a99a" (UID: "af524bf1-8068-45ed-bb37-832eaea6a99a"). InnerVolumeSpecName "kube-api-access-cq29b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:51.461894 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.461870    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"
Oct 30 19:12:51.508642 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.508605    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aaab9a7-bb44-4ab0-813d-03be5fa24219-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aaab9a7-bb44-4ab0-813d-03be5fa24219-plugins-config-volume\") pod \"0aaab9a7-bb44-4ab0-813d-03be5fa24219\" (UID: \"0aaab9a7-bb44-4ab0-813d-03be5fa24219\") "
Oct 30 19:12:51.508813 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.508796    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aaab9a7-bb44-4ab0-813d-03be5fa24219-kube-api-access-7vf8l\" (UniqueName: \"kubernetes.io/projected/0aaab9a7-bb44-4ab0-813d-03be5fa24219-kube-api-access-7vf8l\") pod \"0aaab9a7-bb44-4ab0-813d-03be5fa24219\" (UID: \"0aaab9a7-bb44-4ab0-813d-03be5fa24219\") "
Oct 30 19:12:51.509087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.509059    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aaab9a7-bb44-4ab0-813d-03be5fa24219-plugins-config-volume" pod "0aaab9a7-bb44-4ab0-813d-03be5fa24219" (UID: "0aaab9a7-bb44-4ab0-813d-03be5fa24219"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:51.509665 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.509635    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq29b\" (UniqueName: \"kubernetes.io/projected/af524bf1-8068-45ed-bb37-832eaea6a99a-kube-api-access-cq29b\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:51.509665 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.509665    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af524bf1-8068-45ed-bb37-832eaea6a99a-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:51.509835 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.509681    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aaab9a7-bb44-4ab0-813d-03be5fa24219-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:51.511461 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.511438    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aaab9a7-bb44-4ab0-813d-03be5fa24219-kube-api-access-7vf8l" pod "0aaab9a7-bb44-4ab0-813d-03be5fa24219" (UID: "0aaab9a7-bb44-4ab0-813d-03be5fa24219"). InnerVolumeSpecName "kube-api-access-7vf8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:51.612116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.612040    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vf8l\" (UniqueName: \"kubernetes.io/projected/0aaab9a7-bb44-4ab0-813d-03be5fa24219-kube-api-access-7vf8l\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:51.707229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.707189    6014 generic.go:356] "Generic (PLEG): container finished" podID="af524bf1-8068-45ed-bb37-832eaea6a99a" containerID="fbdf2ed25b57374782f52790755395435905667cb694d8be1a4eaa1133649bf0" exitCode=0
Oct 30 19:12:51.707443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.707234    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"
Oct 30 19:12:51.707443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.707272    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s" event={"ID":"af524bf1-8068-45ed-bb37-832eaea6a99a","Type":"ContainerDied","Data":"fbdf2ed25b57374782f52790755395435905667cb694d8be1a4eaa1133649bf0"}
Oct 30 19:12:51.707443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.707322    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s" event={"ID":"af524bf1-8068-45ed-bb37-832eaea6a99a","Type":"ContainerDied","Data":"c3de788fb4b76bf3deade363f806bb4c749c54930797d6db65090f256f7bd8a4"}
Oct 30 19:12:51.707443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.707349    6014 scope.go:122] "RemoveContainer" containerID="fbdf2ed25b57374782f52790755395435905667cb694d8be1a4eaa1133649bf0"
Oct 30 19:12:51.708958 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.708841    6014 generic.go:356] "Generic (PLEG): container finished" podID="0aaab9a7-bb44-4ab0-813d-03be5fa24219" containerID="b1962d2ab216fefea938f50ac798e0910b4f730eea3ecf700f5a7efc41bab373" exitCode=0
Oct 30 19:12:51.708958 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.708933    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl" event={"ID":"0aaab9a7-bb44-4ab0-813d-03be5fa24219","Type":"ContainerDied","Data":"b1962d2ab216fefea938f50ac798e0910b4f730eea3ecf700f5a7efc41bab373"}
Oct 30 19:12:51.709100 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.708966    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl" event={"ID":"0aaab9a7-bb44-4ab0-813d-03be5fa24219","Type":"ContainerDied","Data":"fecdc0f214556e4443bcdcc70b82c7975beb6ccc1a387bb22afb6c933b09b858"}
Oct 30 19:12:51.709100 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.709027    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"
Oct 30 19:12:51.719079 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.719060    6014 scope.go:122] "RemoveContainer" containerID="fbdf2ed25b57374782f52790755395435905667cb694d8be1a4eaa1133649bf0"
Oct 30 19:12:51.719422 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:12:51.719402    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbdf2ed25b57374782f52790755395435905667cb694d8be1a4eaa1133649bf0\": container with ID starting with fbdf2ed25b57374782f52790755395435905667cb694d8be1a4eaa1133649bf0 not found: ID does not exist" containerID="fbdf2ed25b57374782f52790755395435905667cb694d8be1a4eaa1133649bf0"
Oct 30 19:12:51.719486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.719431    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdf2ed25b57374782f52790755395435905667cb694d8be1a4eaa1133649bf0"} err="failed to get container status \"fbdf2ed25b57374782f52790755395435905667cb694d8be1a4eaa1133649bf0\": rpc error: code = NotFound desc = could not find container \"fbdf2ed25b57374782f52790755395435905667cb694d8be1a4eaa1133649bf0\": container with ID starting with fbdf2ed25b57374782f52790755395435905667cb694d8be1a4eaa1133649bf0 not found: ID does not exist"
Oct 30 19:12:51.719486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.719450    6014 scope.go:122] "RemoveContainer" containerID="b1962d2ab216fefea938f50ac798e0910b4f730eea3ecf700f5a7efc41bab373"
Oct 30 19:12:51.729445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.729424    6014 scope.go:122] "RemoveContainer" containerID="b1962d2ab216fefea938f50ac798e0910b4f730eea3ecf700f5a7efc41bab373"
Oct 30 19:12:51.729717 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:12:51.729698    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1962d2ab216fefea938f50ac798e0910b4f730eea3ecf700f5a7efc41bab373\": container with ID starting with b1962d2ab216fefea938f50ac798e0910b4f730eea3ecf700f5a7efc41bab373 not found: ID does not exist" containerID="b1962d2ab216fefea938f50ac798e0910b4f730eea3ecf700f5a7efc41bab373"
Oct 30 19:12:51.729764 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.729725    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1962d2ab216fefea938f50ac798e0910b4f730eea3ecf700f5a7efc41bab373"} err="failed to get container status \"b1962d2ab216fefea938f50ac798e0910b4f730eea3ecf700f5a7efc41bab373\": rpc error: code = NotFound desc = could not find container \"b1962d2ab216fefea938f50ac798e0910b4f730eea3ecf700f5a7efc41bab373\": container with ID starting with b1962d2ab216fefea938f50ac798e0910b4f730eea3ecf700f5a7efc41bab373 not found: ID does not exist"
Oct 30 19:12:51.871332 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.871244    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"]
Oct 30 19:12:51.874519 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.874499    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2e004a-c7994bdf7-czl2s"]
Oct 30 19:12:51.994306 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:51.994262    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"]
Oct 30 19:12:52.000481 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:52.000451    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl"]
Oct 30 19:12:52.413691 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:52.413656    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e004a-58f566f467-7q7fl" podUID="0aaab9a7-bb44-4ab0-813d-03be5fa24219" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.147:50052\" within 1s: context deadline exceeded"
Oct 30 19:12:52.414111 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:12:52.413758    6014 logging.go:55] [core] [Channel #4631 SubChannel #4632]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.147:50052", ServerName: "10.134.0.147:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.147:50052: operation was canceled"
Oct 30 19:12:53.263404 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:12:53.263356    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:12:53.491216 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:53.491101    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aaab9a7-bb44-4ab0-813d-03be5fa24219" path="/var/lib/kubelet/pods/0aaab9a7-bb44-4ab0-813d-03be5fa24219/volumes"
Oct 30 19:12:53.492646 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:12:53.492622    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af524bf1-8068-45ed-bb37-832eaea6a99a" path="/var/lib/kubelet/pods/af524bf1-8068-45ed-bb37-832eaea6a99a/volumes"
Oct 30 19:13:08.967833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:08.967798    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"]
Oct 30 19:13:08.968574 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:08.968559    6014 cpu_manager.go:414] "containerMap: removing container" podUID="75587314-c5fa-4ce1-8a13-dced50dde605" containerName="payload-pre-processing"
Oct 30 19:13:08.968621 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:08.968576    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75587314-c5fa-4ce1-8a13-dced50dde605" containerName="payload-pre-processing"
Oct 30 19:13:08.968621 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:08.968595    6014 cpu_manager.go:414] "containerMap: removing container" podUID="af524bf1-8068-45ed-bb37-832eaea6a99a" containerName="payload-processing"
Oct 30 19:13:08.968621 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:08.968601    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af524bf1-8068-45ed-bb37-832eaea6a99a" containerName="payload-processing"
Oct 30 19:13:08.968721 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:08.968641    6014 cpu_manager.go:414] "containerMap: removing container" podUID="0aaab9a7-bb44-4ab0-813d-03be5fa24219" containerName="payload-pre-processing"
Oct 30 19:13:08.968721 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:08.968646    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aaab9a7-bb44-4ab0-813d-03be5fa24219" containerName="payload-pre-processing"
Oct 30 19:13:08.968783 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:08.968751    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="75587314-c5fa-4ce1-8a13-dced50dde605" containerName="payload-pre-processing"
Oct 30 19:13:08.968783 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:08.968762    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="af524bf1-8068-45ed-bb37-832eaea6a99a" containerName="payload-processing"
Oct 30 19:13:08.968783 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:08.968776    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aaab9a7-bb44-4ab0-813d-03be5fa24219" containerName="payload-pre-processing"
Oct 30 19:13:08.974253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:08.974226    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.030491 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.030455    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"]
Oct 30 19:13:09.075593 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.075557    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-903a82-serving-cert\""
Oct 30 19:13:09.130363 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.130325    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-maas-api-tls\") pod \"maas-api-e2e-api-b-903a82-5d6794879b-xs8vv\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.130363 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.130369    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7tdp\" (UniqueName: \"kubernetes.io/projected/855f3c98-2ab1-47ad-883e-85410ae82a77-kube-api-access-q7tdp\") pod \"maas-api-e2e-api-b-903a82-5d6794879b-xs8vv\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.130625 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.130561    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/855f3c98-2ab1-47ad-883e-85410ae82a77-openshift-service-ca\") pod \"maas-api-e2e-api-b-903a82-5d6794879b-xs8vv\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.130625 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.130614    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-metrics-certs\") pod \"maas-api-e2e-api-b-903a82-5d6794879b-xs8vv\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.232800 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.232686    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-maas-api-tls\") pod \"maas-api-e2e-api-b-903a82-5d6794879b-xs8vv\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.232800 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.232755    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7tdp\" (UniqueName: \"kubernetes.io/projected/855f3c98-2ab1-47ad-883e-85410ae82a77-kube-api-access-q7tdp\") pod \"maas-api-e2e-api-b-903a82-5d6794879b-xs8vv\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.233069 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.232904    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/855f3c98-2ab1-47ad-883e-85410ae82a77-openshift-service-ca\") pod \"maas-api-e2e-api-b-903a82-5d6794879b-xs8vv\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.233069 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.232986    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-metrics-certs\") pod \"maas-api-e2e-api-b-903a82-5d6794879b-xs8vv\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.233841 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:13:09.233819    6014 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-serving-cert: secret "maas-api-e2e-api-b-903a82-serving-cert" not found
Oct 30 19:13:09.234088 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:13:09.234064    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-maas-api-tls podName:855f3c98-2ab1-47ad-883e-85410ae82a77 nodeName:}" failed. No retries permitted until 2026-10-30 19:13:09.734039402 +0000 UTC m=+2028.377206995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-maas-api-tls") pod "maas-api-e2e-api-b-903a82-5d6794879b-xs8vv" (UID: "855f3c98-2ab1-47ad-883e-85410ae82a77") : secret "maas-api-e2e-api-b-903a82-serving-cert" not found
Oct 30 19:13:09.234554 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.234487    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/855f3c98-2ab1-47ad-883e-85410ae82a77-openshift-service-ca\") pod \"maas-api-e2e-api-b-903a82-5d6794879b-xs8vv\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.236804 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.236775    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-metrics-certs\") pod \"maas-api-e2e-api-b-903a82-5d6794879b-xs8vv\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.293049 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.293016    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7tdp\" (UniqueName: \"kubernetes.io/projected/855f3c98-2ab1-47ad-883e-85410ae82a77-kube-api-access-q7tdp\") pod \"maas-api-e2e-api-b-903a82-5d6794879b-xs8vv\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.518936 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.518847    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"]
Oct 30 19:13:09.519115 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.519049    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx" podUID="4970c31f-3fb0-4265-8e83-94c3f0fa1b99" containerName="istio-proxy" containerID="cri-o://eefe9595ecbfab0ed00d68dba353073200e58d77bb47d84acb25ecac3d8fd08c" gracePeriod=30
Oct 30 19:13:09.601965 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.601908    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:09.602134 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.601983    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx" podUID="4970c31f-3fb0-4265-8e83-94c3f0fa1b99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:09.666927 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.666888    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"]
Oct 30 19:13:09.667112 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.667086    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" podUID="58115155-061f-40f8-b45d-ac30c33b5f2c" containerName="istio-proxy" containerID="cri-o://0dfc5a9bcef0d59949e64cf3f16172b6889ea1f1531fd61b41b044c2e6fedcd8" gracePeriod=30
Oct 30 19:13:09.748928 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.748893    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-maas-api-tls\") pod \"maas-api-e2e-api-b-903a82-5d6794879b-xs8vv\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.751675 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.751645    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-maas-api-tls\") pod \"maas-api-e2e-api-b-903a82-5d6794879b-xs8vv\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:09.890297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:09.890266    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:10.277971 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:10.277943    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"]
Oct 30 19:13:10.279279 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:13:10.279234    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855f3c98_2ab1_47ad_883e_85410ae82a77.slice/crio-f0c378b4ab850873f5207f1c16512ec75436eba2edad42bd4f295ea12d0f80c7 WatchSource:0}: Error finding container f0c378b4ab850873f5207f1c16512ec75436eba2edad42bd4f295ea12d0f80c7: Status 404 returned error can't find the container with id f0c378b4ab850873f5207f1c16512ec75436eba2edad42bd4f295ea12d0f80c7
Oct 30 19:13:10.589688 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:10.589649    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:13:10.863398 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:10.863296    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv" event={"ID":"855f3c98-2ab1-47ad-883e-85410ae82a77","Type":"ContainerStarted","Data":"f0c378b4ab850873f5207f1c16512ec75436eba2edad42bd4f295ea12d0f80c7"}
Oct 30 19:13:11.189715 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:11.189621    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:11.189906 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:11.189704    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" podUID="58115155-061f-40f8-b45d-ac30c33b5f2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:11.872951 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:11.872909    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv" event={"ID":"855f3c98-2ab1-47ad-883e-85410ae82a77","Type":"ContainerStarted","Data":"6343e632ee23b290d9349d6398d09dddba9f97a1a2df53dd0caf28ffcf3a8603"}
Oct 30 19:13:11.873403 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:11.873040    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:12.006356 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:12.006297    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv" podStartSLOduration=3.096314988 podStartE2EDuration="4.006281767s" totalImagesPullingTime="909.966779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:12.003120874 +0000 UTC m=+2030.646288487" watchObservedRunningTime="2026-10-30 19:13:12.006281767 +0000 UTC m=+2030.649449383"
Oct 30 19:13:13.496300 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.496267    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"]
Oct 30 19:13:13.496718 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.496450    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr" podUID="55f0fbaa-51ec-407a-bf9d-d14a2bc93156" containerName="maas-api" containerID="cri-o://2107b03b51d4d7ce3b761a7c20ea3bc0ca7172d63294fd15a9cab7d0e1c35eb9" gracePeriod=30
Oct 30 19:13:13.822880 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.822845    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:13:13.908195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.907733    6014 generic.go:356] "Generic (PLEG): container finished" podID="55f0fbaa-51ec-407a-bf9d-d14a2bc93156" containerID="2107b03b51d4d7ce3b761a7c20ea3bc0ca7172d63294fd15a9cab7d0e1c35eb9" exitCode=0
Oct 30 19:13:13.908195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.907982    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr" event={"ID":"55f0fbaa-51ec-407a-bf9d-d14a2bc93156","Type":"ContainerDied","Data":"2107b03b51d4d7ce3b761a7c20ea3bc0ca7172d63294fd15a9cab7d0e1c35eb9"}
Oct 30 19:13:13.908195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.908064    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr" event={"ID":"55f0fbaa-51ec-407a-bf9d-d14a2bc93156","Type":"ContainerDied","Data":"f943dbb9d533431edc48ce309376e2d8394d273f93dad7144ff678ca271448cd"}
Oct 30 19:13:13.908195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.908105    6014 scope.go:122] "RemoveContainer" containerID="2107b03b51d4d7ce3b761a7c20ea3bc0ca7172d63294fd15a9cab7d0e1c35eb9"
Oct 30 19:13:13.908694 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.908284    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"
Oct 30 19:13:13.920782 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.920756    6014 scope.go:122] "RemoveContainer" containerID="2107b03b51d4d7ce3b761a7c20ea3bc0ca7172d63294fd15a9cab7d0e1c35eb9"
Oct 30 19:13:13.921131 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:13:13.921103    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2107b03b51d4d7ce3b761a7c20ea3bc0ca7172d63294fd15a9cab7d0e1c35eb9\": container with ID starting with 2107b03b51d4d7ce3b761a7c20ea3bc0ca7172d63294fd15a9cab7d0e1c35eb9 not found: ID does not exist" containerID="2107b03b51d4d7ce3b761a7c20ea3bc0ca7172d63294fd15a9cab7d0e1c35eb9"
Oct 30 19:13:13.921131 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.921160    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2107b03b51d4d7ce3b761a7c20ea3bc0ca7172d63294fd15a9cab7d0e1c35eb9"} err="failed to get container status \"2107b03b51d4d7ce3b761a7c20ea3bc0ca7172d63294fd15a9cab7d0e1c35eb9\": rpc error: code = NotFound desc = could not find container \"2107b03b51d4d7ce3b761a7c20ea3bc0ca7172d63294fd15a9cab7d0e1c35eb9\": container with ID starting with 2107b03b51d4d7ce3b761a7c20ea3bc0ca7172d63294fd15a9cab7d0e1c35eb9 not found: ID does not exist"
Oct 30 19:13:13.947523 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.947493    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-kube-api-access-skl82\" (UniqueName: \"kubernetes.io/projected/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-kube-api-access-skl82\") pod \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") "
Oct 30 19:13:13.947749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.947732    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-metrics-certs\") pod \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") "
Oct 30 19:13:13.947828 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.947787    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-maas-api-tls\") pod \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") "
Oct 30 19:13:13.947985 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.947969    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-openshift-service-ca\") pod \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\" (UID: \"55f0fbaa-51ec-407a-bf9d-d14a2bc93156\") "
Oct 30 19:13:13.948357 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.948332    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-openshift-service-ca" pod "55f0fbaa-51ec-407a-bf9d-d14a2bc93156" (UID: "55f0fbaa-51ec-407a-bf9d-d14a2bc93156"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:13.948622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.948605    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:13.949913 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.949891    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-metrics-certs" pod "55f0fbaa-51ec-407a-bf9d-d14a2bc93156" (UID: "55f0fbaa-51ec-407a-bf9d-d14a2bc93156"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:13.950611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.950471    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-kube-api-access-skl82" pod "55f0fbaa-51ec-407a-bf9d-d14a2bc93156" (UID: "55f0fbaa-51ec-407a-bf9d-d14a2bc93156"). InnerVolumeSpecName "kube-api-access-skl82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:13.950611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:13.950548    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-maas-api-tls" pod "55f0fbaa-51ec-407a-bf9d-d14a2bc93156" (UID: "55f0fbaa-51ec-407a-bf9d-d14a2bc93156"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:14.050728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:14.050631    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skl82\" (UniqueName: \"kubernetes.io/projected/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-kube-api-access-skl82\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:14.050728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:14.050671    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:14.050728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:14.050684    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f0fbaa-51ec-407a-bf9d-d14a2bc93156-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:14.330576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:14.330544    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"]
Oct 30 19:13:14.380103 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:14.380060    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b8d0f-588cd55bb8-m7dsr"]
Oct 30 19:13:14.671545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:14.671500    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"]
Oct 30 19:13:14.672030 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:14.671800    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj" podUID="d55d32d8-2bc9-4a98-98c1-a8701348f81b" containerName="maas-api" containerID="cri-o://7b16633ae8c24f809f8805996192940aa3b68d425fbbcfbea03493aa76bb3795" gracePeriod=30
Oct 30 19:13:14.976128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:14.974782    6014 generic.go:356] "Generic (PLEG): container finished" podID="58115155-061f-40f8-b45d-ac30c33b5f2c" containerID="0dfc5a9bcef0d59949e64cf3f16172b6889ea1f1531fd61b41b044c2e6fedcd8" exitCode=0
Oct 30 19:13:14.976128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:14.974925    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" event={"ID":"58115155-061f-40f8-b45d-ac30c33b5f2c","Type":"ContainerDied","Data":"0dfc5a9bcef0d59949e64cf3f16172b6889ea1f1531fd61b41b044c2e6fedcd8"}
Oct 30 19:13:14.983612 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:14.980488    6014 generic.go:356] "Generic (PLEG): container finished" podID="4970c31f-3fb0-4265-8e83-94c3f0fa1b99" containerID="eefe9595ecbfab0ed00d68dba353073200e58d77bb47d84acb25ecac3d8fd08c" exitCode=0
Oct 30 19:13:14.983612 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:14.980573    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx" event={"ID":"4970c31f-3fb0-4265-8e83-94c3f0fa1b99","Type":"ContainerDied","Data":"eefe9595ecbfab0ed00d68dba353073200e58d77bb47d84acb25ecac3d8fd08c"}
Oct 30 19:13:14.983612 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:14.982915    6014 generic.go:356] "Generic (PLEG): container finished" podID="d55d32d8-2bc9-4a98-98c1-a8701348f81b" containerID="7b16633ae8c24f809f8805996192940aa3b68d425fbbcfbea03493aa76bb3795" exitCode=0
Oct 30 19:13:14.983612 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:14.982972    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj" event={"ID":"d55d32d8-2bc9-4a98-98c1-a8701348f81b","Type":"ContainerDied","Data":"7b16633ae8c24f809f8805996192940aa3b68d425fbbcfbea03493aa76bb3795"}
Oct 30 19:13:15.023160 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.023112    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 19:13:15.027835 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.027752    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115042    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-podinfo\") pod \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115104    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-socket\") pod \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115194    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-data\") pod \"58115155-061f-40f8-b45d-ac30c33b5f2c\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115241    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-podinfo\") pod \"58115155-061f-40f8-b45d-ac30c33b5f2c\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115276    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-certs\") pod \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115332    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-token\" (UniqueName: \"kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-token\") pod \"58115155-061f-40f8-b45d-ac30c33b5f2c\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115384    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58115155-061f-40f8-b45d-ac30c33b5f2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58115155-061f-40f8-b45d-ac30c33b5f2c-istiod-ca-cert\") pod \"58115155-061f-40f8-b45d-ac30c33b5f2c\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115437    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-credential-socket\") pod \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115514    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-certs\") pod \"58115155-061f-40f8-b45d-ac30c33b5f2c\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115545    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-kube-api-access-9hdmg\" (UniqueName: \"kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-kube-api-access-9hdmg\") pod \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115645    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-envoy\") pod \"58115155-061f-40f8-b45d-ac30c33b5f2c\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115777    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-data\") pod \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115825    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-envoy\") pod \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115857    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-credential-socket\") pod \"58115155-061f-40f8-b45d-ac30c33b5f2c\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") "
Oct 30 19:13:15.125165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.115906    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-kube-api-access-vf8tv\" (UniqueName: \"kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-kube-api-access-vf8tv\") pod \"58115155-061f-40f8-b45d-ac30c33b5f2c\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") "
Oct 30 19:13:15.141312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.136550    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-data" pod "58115155-061f-40f8-b45d-ac30c33b5f2c" (UID: "58115155-061f-40f8-b45d-ac30c33b5f2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.141312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.136635    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58115155-061f-40f8-b45d-ac30c33b5f2c-istiod-ca-cert" pod "58115155-061f-40f8-b45d-ac30c33b5f2c" (UID: "58115155-061f-40f8-b45d-ac30c33b5f2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:15.141312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.141115    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-certs" pod "58115155-061f-40f8-b45d-ac30c33b5f2c" (UID: "58115155-061f-40f8-b45d-ac30c33b5f2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.141618 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.141375    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-credential-socket" pod "4970c31f-3fb0-4265-8e83-94c3f0fa1b99" (UID: "4970c31f-3fb0-4265-8e83-94c3f0fa1b99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.148314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.147884    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-certs" pod "4970c31f-3fb0-4265-8e83-94c3f0fa1b99" (UID: "4970c31f-3fb0-4265-8e83-94c3f0fa1b99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.148314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.148162    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-socket" pod "4970c31f-3fb0-4265-8e83-94c3f0fa1b99" (UID: "4970c31f-3fb0-4265-8e83-94c3f0fa1b99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.148749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.148574    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-credential-socket" pod "58115155-061f-40f8-b45d-ac30c33b5f2c" (UID: "58115155-061f-40f8-b45d-ac30c33b5f2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.148893 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.148811    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-data" pod "4970c31f-3fb0-4265-8e83-94c3f0fa1b99" (UID: "4970c31f-3fb0-4265-8e83-94c3f0fa1b99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.149300 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.149274    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-socket\") pod \"58115155-061f-40f8-b45d-ac30c33b5f2c\" (UID: \"58115155-061f-40f8-b45d-ac30c33b5f2c\") "
Oct 30 19:13:15.149397 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.149331    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istiod-ca-cert\") pod \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") "
Oct 30 19:13:15.149965 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.149928    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-token\" (UniqueName: \"kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-token\") pod \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\" (UID: \"4970c31f-3fb0-4265-8e83-94c3f0fa1b99\") "
Oct 30 19:13:15.151117 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.151087    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58115155-061f-40f8-b45d-ac30c33b5f2c-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.151243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.151123    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.151243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.151158    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.151243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.151173    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.151243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.151187    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.151243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.151202    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.151243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.151215    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.151243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.151229    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.152177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.151870    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-kube-api-access-9hdmg" pod "4970c31f-3fb0-4265-8e83-94c3f0fa1b99" (UID: "4970c31f-3fb0-4265-8e83-94c3f0fa1b99"). InnerVolumeSpecName "kube-api-access-9hdmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:15.152177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.152073    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-socket" pod "58115155-061f-40f8-b45d-ac30c33b5f2c" (UID: "58115155-061f-40f8-b45d-ac30c33b5f2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.152177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.152132    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istiod-ca-cert" pod "4970c31f-3fb0-4265-8e83-94c3f0fa1b99" (UID: "4970c31f-3fb0-4265-8e83-94c3f0fa1b99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:15.152650 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.152615    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-envoy" pod "58115155-061f-40f8-b45d-ac30c33b5f2c" (UID: "58115155-061f-40f8-b45d-ac30c33b5f2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.154984 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.154964    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-token" pod "4970c31f-3fb0-4265-8e83-94c3f0fa1b99" (UID: "4970c31f-3fb0-4265-8e83-94c3f0fa1b99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:15.155088 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.155018    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-token" pod "58115155-061f-40f8-b45d-ac30c33b5f2c" (UID: "58115155-061f-40f8-b45d-ac30c33b5f2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:15.155088 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.155061    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-podinfo" pod "4970c31f-3fb0-4265-8e83-94c3f0fa1b99" (UID: "4970c31f-3fb0-4265-8e83-94c3f0fa1b99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:15.155283 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.155096    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-podinfo" pod "58115155-061f-40f8-b45d-ac30c33b5f2c" (UID: "58115155-061f-40f8-b45d-ac30c33b5f2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:15.155790 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.155770    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-envoy" pod "4970c31f-3fb0-4265-8e83-94c3f0fa1b99" (UID: "4970c31f-3fb0-4265-8e83-94c3f0fa1b99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.156680 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.156659    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-kube-api-access-vf8tv" pod "58115155-061f-40f8-b45d-ac30c33b5f2c" (UID: "58115155-061f-40f8-b45d-ac30c33b5f2c"). InnerVolumeSpecName "kube-api-access-vf8tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:15.163780 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.163756    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:13:15.253015 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.252975    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-metrics-certs\") pod \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") "
Oct 30 19:13:15.253376 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.253358    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-maas-api-tls\") pod \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") "
Oct 30 19:13:15.253545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.253411    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d55d32d8-2bc9-4a98-98c1-a8701348f81b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55d32d8-2bc9-4a98-98c1-a8701348f81b-openshift-service-ca\") pod \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") "
Oct 30 19:13:15.253545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.253511    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d55d32d8-2bc9-4a98-98c1-a8701348f81b-kube-api-access-j5ghv\" (UniqueName: \"kubernetes.io/projected/d55d32d8-2bc9-4a98-98c1-a8701348f81b-kube-api-access-j5ghv\") pod \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\" (UID: \"d55d32d8-2bc9-4a98-98c1-a8701348f81b\") "
Oct 30 19:13:15.254304 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.254278    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.254304 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.254305    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.254540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.254319    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf8tv\" (UniqueName: \"kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-kube-api-access-vf8tv\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.254540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.254333    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58115155-061f-40f8-b45d-ac30c33b5f2c-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.254540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.254347    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.254540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.254361    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.254540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.254373    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.254540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.254386    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.254540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.254399    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58115155-061f-40f8-b45d-ac30c33b5f2c-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.254540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.254413    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hdmg\" (UniqueName: \"kubernetes.io/projected/4970c31f-3fb0-4265-8e83-94c3f0fa1b99-kube-api-access-9hdmg\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.254964 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.254729    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55d32d8-2bc9-4a98-98c1-a8701348f81b-openshift-service-ca" pod "d55d32d8-2bc9-4a98-98c1-a8701348f81b" (UID: "d55d32d8-2bc9-4a98-98c1-a8701348f81b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:15.259261 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.259232    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-maas-api-tls" pod "d55d32d8-2bc9-4a98-98c1-a8701348f81b" (UID: "d55d32d8-2bc9-4a98-98c1-a8701348f81b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:15.259976 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.259951    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55d32d8-2bc9-4a98-98c1-a8701348f81b-kube-api-access-j5ghv" pod "d55d32d8-2bc9-4a98-98c1-a8701348f81b" (UID: "d55d32d8-2bc9-4a98-98c1-a8701348f81b"). InnerVolumeSpecName "kube-api-access-j5ghv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:15.260076 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.260048    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-metrics-certs" pod "d55d32d8-2bc9-4a98-98c1-a8701348f81b" (UID: "d55d32d8-2bc9-4a98-98c1-a8701348f81b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:15.357317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.357284    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.357317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.357313    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55d32d8-2bc9-4a98-98c1-a8701348f81b-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.357317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.357321    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5ghv\" (UniqueName: \"kubernetes.io/projected/d55d32d8-2bc9-4a98-98c1-a8701348f81b-kube-api-access-j5ghv\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.357637 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.357330    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55d32d8-2bc9-4a98-98c1-a8701348f81b-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.478398 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.478318    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f0fbaa-51ec-407a-bf9d-d14a2bc93156" path="/var/lib/kubelet/pods/55f0fbaa-51ec-407a-bf9d-d14a2bc93156/volumes"
Oct 30 19:13:15.993668 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.993636    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"
Oct 30 19:13:15.993668 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.993649    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx" event={"ID":"4970c31f-3fb0-4265-8e83-94c3f0fa1b99","Type":"ContainerDied","Data":"f8a958b22eb7914c8b4dd2174d2b214eff9a810b2c99c7fee9b42ec699e31857"}
Oct 30 19:13:15.994276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.993719    6014 scope.go:122] "RemoveContainer" containerID="eefe9595ecbfab0ed00d68dba353073200e58d77bb47d84acb25ecac3d8fd08c"
Oct 30 19:13:15.995848 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.995818    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj" event={"ID":"d55d32d8-2bc9-4a98-98c1-a8701348f81b","Type":"ContainerDied","Data":"1070593a5736ce515a4c49895b379780f52b6ee441b05d2bbbe8a96d8b659fed"}
Oct 30 19:13:15.995974 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:15.995887    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"
Oct 30 19:13:16.002274 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:16.001731    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r" event={"ID":"58115155-061f-40f8-b45d-ac30c33b5f2c","Type":"ContainerDied","Data":"63d1a3b797e3e7ec4c1e6dba0d2103a6d2c5a4e72ae3d64b1e8375987379754b"}
Oct 30 19:13:16.002274 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:16.001749    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"
Oct 30 19:13:16.011641 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:16.011623    6014 scope.go:122] "RemoveContainer" containerID="7b16633ae8c24f809f8805996192940aa3b68d425fbbcfbea03493aa76bb3795"
Oct 30 19:13:16.021285 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:16.021266    6014 scope.go:122] "RemoveContainer" containerID="0dfc5a9bcef0d59949e64cf3f16172b6889ea1f1531fd61b41b044c2e6fedcd8"
Oct 30 19:13:16.162615 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:16.162579    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"]
Oct 30 19:13:16.165924 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:16.165893    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae6a138d-d4cf57f4b-v8xfj"]
Oct 30 19:13:16.275582 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:16.275496    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"]
Oct 30 19:13:16.285894 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:16.285860    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-fz7wx"]
Oct 30 19:13:16.393749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:16.393714    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"]
Oct 30 19:13:16.399588 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:16.399552    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-g5b2r"]
Oct 30 19:13:17.479760 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:17.479722    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4970c31f-3fb0-4265-8e83-94c3f0fa1b99" path="/var/lib/kubelet/pods/4970c31f-3fb0-4265-8e83-94c3f0fa1b99/volumes"
Oct 30 19:13:17.480515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:17.480497    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58115155-061f-40f8-b45d-ac30c33b5f2c" path="/var/lib/kubelet/pods/58115155-061f-40f8-b45d-ac30c33b5f2c/volumes"
Oct 30 19:13:17.481067 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:17.481051    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d55d32d8-2bc9-4a98-98c1-a8701348f81b" path="/var/lib/kubelet/pods/d55d32d8-2bc9-4a98-98c1-a8701348f81b/volumes"
Oct 30 19:13:17.886042 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:17.886015    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:13:22.955739 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:22.955655    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"]
Oct 30 19:13:22.956218 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:22.955886    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerName="istio-proxy" containerID="cri-o://c3637d43263f335f5006257860d004e8e4137bcb26b654296e8123ca0b0766ab" gracePeriod=30
Oct 30 19:13:23.260537 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:13:23.260445    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:13:24.335389 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:24.335350    6014 scope.go:122] "RemoveContainer" containerID="70fd3e1b9d129de52b35dff9ec8a7ca0e665dc040daa68876b9103188491ddc2"
Oct 30 19:13:27.650442 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:27.650402    6014 patch_prober.go:29] interesting pod/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:27.650876 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:27.650479    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:28.164751 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.164717    6014 generic.go:356] "Generic (PLEG): container finished" podID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerID="c3637d43263f335f5006257860d004e8e4137bcb26b654296e8123ca0b0766ab" exitCode=0
Oct 30 19:13:28.164937 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.164795    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" event={"ID":"07a2f9fa-4621-45b8-84a8-2aa9721df791","Type":"ContainerDied","Data":"c3637d43263f335f5006257860d004e8e4137bcb26b654296e8123ca0b0766ab"}
Oct 30 19:13:28.164937 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.164888    6014 scope.go:122] "RemoveContainer" containerID="2e7bbe2992568a048be8347dfa6ed2188d3b0ee8b67ef2d45d6f28630e0ef7a2"
Oct 30 19:13:28.331482 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.331444    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"]
Oct 30 19:13:28.332243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.332224    6014 cpu_manager.go:414] "containerMap: removing container" podUID="4970c31f-3fb0-4265-8e83-94c3f0fa1b99" containerName="istio-proxy"
Oct 30 19:13:28.332304 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.332245    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4970c31f-3fb0-4265-8e83-94c3f0fa1b99" containerName="istio-proxy"
Oct 30 19:13:28.332304 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.332266    6014 cpu_manager.go:414] "containerMap: removing container" podUID="55f0fbaa-51ec-407a-bf9d-d14a2bc93156" containerName="maas-api"
Oct 30 19:13:28.332304 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.332271    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f0fbaa-51ec-407a-bf9d-d14a2bc93156" containerName="maas-api"
Oct 30 19:13:28.332304 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.332287    6014 cpu_manager.go:414] "containerMap: removing container" podUID="d55d32d8-2bc9-4a98-98c1-a8701348f81b" containerName="maas-api"
Oct 30 19:13:28.332304 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.332293    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55d32d8-2bc9-4a98-98c1-a8701348f81b" containerName="maas-api"
Oct 30 19:13:28.332304 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.332304    6014 cpu_manager.go:414] "containerMap: removing container" podUID="58115155-061f-40f8-b45d-ac30c33b5f2c" containerName="istio-proxy"
Oct 30 19:13:28.332601 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.332309    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58115155-061f-40f8-b45d-ac30c33b5f2c" containerName="istio-proxy"
Oct 30 19:13:28.332601 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.332457    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="4970c31f-3fb0-4265-8e83-94c3f0fa1b99" containerName="istio-proxy"
Oct 30 19:13:28.332601 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.332469    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f0fbaa-51ec-407a-bf9d-d14a2bc93156" containerName="maas-api"
Oct 30 19:13:28.332601 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.332477    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d55d32d8-2bc9-4a98-98c1-a8701348f81b" containerName="maas-api"
Oct 30 19:13:28.332601 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.332486    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="58115155-061f-40f8-b45d-ac30c33b5f2c" containerName="istio-proxy"
Oct 30 19:13:28.337806 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.337779    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"
Oct 30 19:13:28.395707 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.395668    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"]
Oct 30 19:13:28.448642 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.448607    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ae6a138d-dockercfg-2sxmf\""
Oct 30 19:13:28.494162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.489885    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"]
Oct 30 19:13:28.502702 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.502672    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"
Oct 30 19:13:28.516048 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.515912    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ae6a138d\""
Oct 30 19:13:28.525730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.524763    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5eebbf-70d4-48fe-827a-965f191a8938-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6\" (UID: \"ea5eebbf-70d4-48fe-827a-965f191a8938\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"
Oct 30 19:13:28.525730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.525022    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t574\" (UniqueName: \"kubernetes.io/projected/97b68589-f0f6-4d49-b519-9d30746433ca-kube-api-access-5t574\") pod \"payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9\" (UID: \"97b68589-f0f6-4d49-b519-9d30746433ca\") " pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"
Oct 30 19:13:28.525730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.525080    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4fbc\" (UniqueName: \"kubernetes.io/projected/ea5eebbf-70d4-48fe-827a-965f191a8938-kube-api-access-p4fbc\") pod \"payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6\" (UID: \"ea5eebbf-70d4-48fe-827a-965f191a8938\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"
Oct 30 19:13:28.525730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.525220    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b68589-f0f6-4d49-b519-9d30746433ca-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9\" (UID: \"97b68589-f0f6-4d49-b519-9d30746433ca\") " pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"
Oct 30 19:13:28.559980 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.559944    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"]
Oct 30 19:13:28.626476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.626388    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t574\" (UniqueName: \"kubernetes.io/projected/97b68589-f0f6-4d49-b519-9d30746433ca-kube-api-access-5t574\") pod \"payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9\" (UID: \"97b68589-f0f6-4d49-b519-9d30746433ca\") " pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"
Oct 30 19:13:28.626476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.626451    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4fbc\" (UniqueName: \"kubernetes.io/projected/ea5eebbf-70d4-48fe-827a-965f191a8938-kube-api-access-p4fbc\") pod \"payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6\" (UID: \"ea5eebbf-70d4-48fe-827a-965f191a8938\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"
Oct 30 19:13:28.626710 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.626556    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b68589-f0f6-4d49-b519-9d30746433ca-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9\" (UID: \"97b68589-f0f6-4d49-b519-9d30746433ca\") " pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"
Oct 30 19:13:28.626710 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.626618    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5eebbf-70d4-48fe-827a-965f191a8938-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6\" (UID: \"ea5eebbf-70d4-48fe-827a-965f191a8938\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"
Oct 30 19:13:28.627515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.627492    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b68589-f0f6-4d49-b519-9d30746433ca-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9\" (UID: \"97b68589-f0f6-4d49-b519-9d30746433ca\") " pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"
Oct 30 19:13:28.627641 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.627619    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5eebbf-70d4-48fe-827a-965f191a8938-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6\" (UID: \"ea5eebbf-70d4-48fe-827a-965f191a8938\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"
Oct 30 19:13:28.683827 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.683787    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4fbc\" (UniqueName: \"kubernetes.io/projected/ea5eebbf-70d4-48fe-827a-965f191a8938-kube-api-access-p4fbc\") pod \"payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6\" (UID: \"ea5eebbf-70d4-48fe-827a-965f191a8938\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"
Oct 30 19:13:28.686352 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.686323    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t574\" (UniqueName: \"kubernetes.io/projected/97b68589-f0f6-4d49-b519-9d30746433ca-kube-api-access-5t574\") pod \"payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9\" (UID: \"97b68589-f0f6-4d49-b519-9d30746433ca\") " pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"
Oct 30 19:13:28.742938 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.742910    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:13:28.829504 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.829464    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-certs\") pod \"07a2f9fa-4621-45b8-84a8-2aa9721df791\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") "
Oct 30 19:13:28.829703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.829526    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a2f9fa-4621-45b8-84a8-2aa9721df791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a2f9fa-4621-45b8-84a8-2aa9721df791-istiod-ca-cert\") pod \"07a2f9fa-4621-45b8-84a8-2aa9721df791\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") "
Oct 30 19:13:28.829703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.829554    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-envoy\") pod \"07a2f9fa-4621-45b8-84a8-2aa9721df791\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") "
Oct 30 19:13:28.829703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.829591    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-socket\") pod \"07a2f9fa-4621-45b8-84a8-2aa9721df791\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") "
Oct 30 19:13:28.829703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.829687    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-token\" (UniqueName: \"kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-token\") pod \"07a2f9fa-4621-45b8-84a8-2aa9721df791\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") "
Oct 30 19:13:28.829932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.829722    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-kube-api-access-pr2mn\" (UniqueName: \"kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-kube-api-access-pr2mn\") pod \"07a2f9fa-4621-45b8-84a8-2aa9721df791\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") "
Oct 30 19:13:28.829932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.829752    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-podinfo\") pod \"07a2f9fa-4621-45b8-84a8-2aa9721df791\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") "
Oct 30 19:13:28.829932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.829779    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-data\") pod \"07a2f9fa-4621-45b8-84a8-2aa9721df791\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") "
Oct 30 19:13:28.829932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.829801    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-certs" pod "07a2f9fa-4621-45b8-84a8-2aa9721df791" (UID: "07a2f9fa-4621-45b8-84a8-2aa9721df791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:28.829932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.829830    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-credential-socket\") pod \"07a2f9fa-4621-45b8-84a8-2aa9721df791\" (UID: \"07a2f9fa-4621-45b8-84a8-2aa9721df791\") "
Oct 30 19:13:28.830270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.829992    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-socket" pod "07a2f9fa-4621-45b8-84a8-2aa9721df791" (UID: "07a2f9fa-4621-45b8-84a8-2aa9721df791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:28.830270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.830003    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a2f9fa-4621-45b8-84a8-2aa9721df791-istiod-ca-cert" pod "07a2f9fa-4621-45b8-84a8-2aa9721df791" (UID: "07a2f9fa-4621-45b8-84a8-2aa9721df791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:28.830270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.830226    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-data" pod "07a2f9fa-4621-45b8-84a8-2aa9721df791" (UID: "07a2f9fa-4621-45b8-84a8-2aa9721df791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:28.830538 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.830405    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-credential-socket" pod "07a2f9fa-4621-45b8-84a8-2aa9721df791" (UID: "07a2f9fa-4621-45b8-84a8-2aa9721df791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:28.830790 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.830770    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:28.830834 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.830797    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a2f9fa-4621-45b8-84a8-2aa9721df791-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:28.830834 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.830812    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:28.830834 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.830827    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:28.830960 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.830841    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:28.832200 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.832172    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-envoy" pod "07a2f9fa-4621-45b8-84a8-2aa9721df791" (UID: "07a2f9fa-4621-45b8-84a8-2aa9721df791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:28.832293 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.832203    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-token" pod "07a2f9fa-4621-45b8-84a8-2aa9721df791" (UID: "07a2f9fa-4621-45b8-84a8-2aa9721df791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:28.832689 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.832669    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-podinfo" pod "07a2f9fa-4621-45b8-84a8-2aa9721df791" (UID: "07a2f9fa-4621-45b8-84a8-2aa9721df791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:28.832766 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.832740    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-kube-api-access-pr2mn" pod "07a2f9fa-4621-45b8-84a8-2aa9721df791" (UID: "07a2f9fa-4621-45b8-84a8-2aa9721df791"). InnerVolumeSpecName "kube-api-access-pr2mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:28.872232 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.872195    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"
Oct 30 19:13:28.941617 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.941512    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:28.941617 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.941553    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:28.941617 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.941572    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr2mn\" (UniqueName: \"kubernetes.io/projected/07a2f9fa-4621-45b8-84a8-2aa9721df791-kube-api-access-pr2mn\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:28.941617 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.941589    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a2f9fa-4621-45b8-84a8-2aa9721df791-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:28.969184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:28.968579    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"
Oct 30 19:13:29.174763 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:29.174675    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"
Oct 30 19:13:29.174763 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:29.174701    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8" event={"ID":"07a2f9fa-4621-45b8-84a8-2aa9721df791","Type":"ContainerDied","Data":"ad3d70e6d3bd3eb9907aa807ac69dce2d6acf7c6cb9be943fbd41e4dbaacb9fc"}
Oct 30 19:13:29.174763 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:29.174763    6014 scope.go:122] "RemoveContainer" containerID="c3637d43263f335f5006257860d004e8e4137bcb26b654296e8123ca0b0766ab"
Oct 30 19:13:29.300570 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:29.300541    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"]
Oct 30 19:13:29.301768 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:13:29.301740    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5eebbf_70d4_48fe_827a_965f191a8938.slice/crio-865076b3d870e458d907104367ea1f9b0c07ca71a79ca8791fbacd0f3633aabe WatchSource:0}: Error finding container 865076b3d870e458d907104367ea1f9b0c07ca71a79ca8791fbacd0f3633aabe: Status 404 returned error can't find the container with id 865076b3d870e458d907104367ea1f9b0c07ca71a79ca8791fbacd0f3633aabe
Oct 30 19:13:29.345463 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:29.345427    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"]
Oct 30 19:13:29.350723 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:29.350484    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0b8d0f-openshift-default-666fdf65fc-4pxr8"]
Oct 30 19:13:29.408742 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:29.408719    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"]
Oct 30 19:13:29.410505 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:13:29.410473    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b68589_f0f6_4d49_b519_9d30746433ca.slice/crio-bd814c74cef4053e017e27bf9f045fd2271515b4d17f0c41007d4b699465d34c WatchSource:0}: Error finding container bd814c74cef4053e017e27bf9f045fd2271515b4d17f0c41007d4b699465d34c: Status 404 returned error can't find the container with id bd814c74cef4053e017e27bf9f045fd2271515b4d17f0c41007d4b699465d34c
Oct 30 19:13:29.479097 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:29.479066    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" path="/var/lib/kubelet/pods/07a2f9fa-4621-45b8-84a8-2aa9721df791/volumes"
Oct 30 19:13:30.184992 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:30.184958    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9" event={"ID":"97b68589-f0f6-4d49-b519-9d30746433ca","Type":"ContainerStarted","Data":"8a404236ae55b1b8ac99a51fd7e7315ffa5d364d5709cbac5896816ecee9cffc"}
Oct 30 19:13:30.185532 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:30.185006    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9" event={"ID":"97b68589-f0f6-4d49-b519-9d30746433ca","Type":"ContainerStarted","Data":"bd814c74cef4053e017e27bf9f045fd2271515b4d17f0c41007d4b699465d34c"}
Oct 30 19:13:30.185532 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:30.185192    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"
Oct 30 19:13:30.186749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:30.186724    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6" event={"ID":"ea5eebbf-70d4-48fe-827a-965f191a8938","Type":"ContainerStarted","Data":"7a9be7426a674d37965faca4f3cb50e9036373362f4fe5122f6bbdb5937ad8fc"}
Oct 30 19:13:30.186749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:30.186765    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6" event={"ID":"ea5eebbf-70d4-48fe-827a-965f191a8938","Type":"ContainerStarted","Data":"865076b3d870e458d907104367ea1f9b0c07ca71a79ca8791fbacd0f3633aabe"}
Oct 30 19:13:30.186962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:30.186849    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"
Oct 30 19:13:30.301700 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:30.301652    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9" podStartSLOduration=2.301635776 podStartE2EDuration="2.301635776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:30.297986483 +0000 UTC m=+2048.941154108" watchObservedRunningTime="2026-10-30 19:13:30.301635776 +0000 UTC m=+2048.944803399"
Oct 30 19:13:41.197386 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:41.197350    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"
Oct 30 19:13:41.197896 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:41.197441    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"
Oct 30 19:13:41.313463 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:41.313409    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6" podStartSLOduration=13.313392456 podStartE2EDuration="13.313392456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:30.430446212 +0000 UTC m=+2049.073613828" watchObservedRunningTime="2026-10-30 19:13:41.313392456 +0000 UTC m=+2059.956560073"
Oct 30 19:13:48.277717 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:48.277681    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"]
Oct 30 19:13:53.260171 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:13:53.260062    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:13:53.278534 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.278493    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc" podUID="577ef8da-25fd-49ff-bbd4-dc8e568fc99b" containerName="payload-processing" containerID="cri-o://80988b742ffb21fd116c9977cca65ead6019a5b86aef38ced58d6be572c7c33d" gracePeriod=25
Oct 30 19:13:53.495772 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.495739    6014 generic.go:356] "Generic (PLEG): container finished" podID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerID="62696aee7734892c47050c55661aaac6d53cdba08b03020d45c2355cace1640d" exitCode=0
Oct 30 19:13:53.495947 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.495837    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h" event={"ID":"e693b661-2b77-4bf5-82f7-b11fd1e570bb","Type":"ContainerDied","Data":"62696aee7734892c47050c55661aaac6d53cdba08b03020d45c2355cace1640d"}
Oct 30 19:13:53.496452 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.496419    6014 scope.go:122] "RemoveContainer" containerID="62696aee7734892c47050c55661aaac6d53cdba08b03020d45c2355cace1640d"
Oct 30 19:13:53.497629 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.497609    6014 generic.go:356] "Generic (PLEG): container finished" podID="577ef8da-25fd-49ff-bbd4-dc8e568fc99b" containerID="80988b742ffb21fd116c9977cca65ead6019a5b86aef38ced58d6be572c7c33d" exitCode=0
Oct 30 19:13:53.497765 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.497753    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc" event={"ID":"577ef8da-25fd-49ff-bbd4-dc8e568fc99b","Type":"ContainerDied","Data":"80988b742ffb21fd116c9977cca65ead6019a5b86aef38ced58d6be572c7c33d"}
Oct 30 19:13:53.498999 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.498964    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:13:53.499122 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.499043    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:13:53.499122 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.499106    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:13:53.604703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.604675    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"
Oct 30 19:13:53.669558 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.669460    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-plugins-config-volume\") pod \"577ef8da-25fd-49ff-bbd4-dc8e568fc99b\" (UID: \"577ef8da-25fd-49ff-bbd4-dc8e568fc99b\") "
Oct 30 19:13:53.669558 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.669510    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-kube-api-access-lg9d4\" (UniqueName: \"kubernetes.io/projected/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-kube-api-access-lg9d4\") pod \"577ef8da-25fd-49ff-bbd4-dc8e568fc99b\" (UID: \"577ef8da-25fd-49ff-bbd4-dc8e568fc99b\") "
Oct 30 19:13:53.669938 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.669913    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-plugins-config-volume" pod "577ef8da-25fd-49ff-bbd4-dc8e568fc99b" (UID: "577ef8da-25fd-49ff-bbd4-dc8e568fc99b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:53.670104 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.670082    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:53.672241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.672210    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-kube-api-access-lg9d4" pod "577ef8da-25fd-49ff-bbd4-dc8e568fc99b" (UID: "577ef8da-25fd-49ff-bbd4-dc8e568fc99b"). InnerVolumeSpecName "kube-api-access-lg9d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:53.773914 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:53.773028    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg9d4\" (UniqueName: \"kubernetes.io/projected/577ef8da-25fd-49ff-bbd4-dc8e568fc99b-kube-api-access-lg9d4\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:54.517523 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:54.517486    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h" event={"ID":"e693b661-2b77-4bf5-82f7-b11fd1e570bb","Type":"ContainerStarted","Data":"e68ef00a0d664948bb263145a4c5db8deea3a2b1737b86f1421fc1801e32d2c0"}
Oct 30 19:13:54.519101 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:54.519078    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"
Oct 30 19:13:54.519101 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:54.519091    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc" event={"ID":"577ef8da-25fd-49ff-bbd4-dc8e568fc99b","Type":"ContainerDied","Data":"32e79d5c56eabebdf73fe5c56a0af43c8d17ae5d4255b228ba6ed253a9860d24"}
Oct 30 19:13:54.519332 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:54.519157    6014 scope.go:122] "RemoveContainer" containerID="80988b742ffb21fd116c9977cca65ead6019a5b86aef38ced58d6be572c7c33d"
Oct 30 19:13:54.772222 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:54.762205    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"]
Oct 30 19:13:54.774575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:54.774543    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4ebeba-68d57b7f49-r49mc"]
Oct 30 19:13:54.981850 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:54.981814    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:13:55.431016 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:55.430980    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:13:55.432179 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:55.432122    6014 patch_prober.go:29] interesting pod/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 19:13:55.432270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:55.432208    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h" podUID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 19:13:55.482087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:55.482053    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="577ef8da-25fd-49ff-bbd4-dc8e568fc99b" path="/var/lib/kubelet/pods/577ef8da-25fd-49ff-bbd4-dc8e568fc99b/volumes"
Oct 30 19:13:56.430876 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:56.430836    6014 patch_prober.go:29] interesting pod/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 19:13:56.431344 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:56.430905    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h" podUID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 19:13:57.431093 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:57.431054    6014 patch_prober.go:29] interesting pod/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 19:13:57.431536 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:57.431131    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h" podUID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 19:13:58.431374 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:58.431336    6014 patch_prober.go:29] interesting pod/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 19:13:58.431805 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:58.431401    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h" podUID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 19:13:59.431381 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:59.431341    6014 patch_prober.go:29] interesting pod/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 19:13:59.431805 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:13:59.431421    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h" podUID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 19:14:00.436074 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:00.435995    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:14:00.437330 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:00.437307    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:14:03.479658 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.479624    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"]
Oct 30 19:14:03.480189 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.479862    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b" podUID="97e8b1ea-88f1-4f81-80b1-2b7c81e6926e" containerName="maas-api" containerID="cri-o://fd1d8f3b98dd3b3e080f48ecec733594c66f6727be5ca881166b10309dc988f0" gracePeriod=30
Oct 30 19:14:03.611673 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.611637    6014 generic.go:356] "Generic (PLEG): container finished" podID="97e8b1ea-88f1-4f81-80b1-2b7c81e6926e" containerID="fd1d8f3b98dd3b3e080f48ecec733594c66f6727be5ca881166b10309dc988f0" exitCode=0
Oct 30 19:14:03.611844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.611666    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b" event={"ID":"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e","Type":"ContainerDied","Data":"fd1d8f3b98dd3b3e080f48ecec733594c66f6727be5ca881166b10309dc988f0"}
Oct 30 19:14:03.799767 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.799721    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:14:03.958494 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.958455    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-maas-api-tls\") pod \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") "
Oct 30 19:14:03.958963 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.958939    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-metrics-certs\") pod \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") "
Oct 30 19:14:03.965314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.965281    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-openshift-service-ca\") pod \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") "
Oct 30 19:14:03.972788 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.965638    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-openshift-service-ca" pod "97e8b1ea-88f1-4f81-80b1-2b7c81e6926e" (UID: "97e8b1ea-88f1-4f81-80b1-2b7c81e6926e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:03.972788 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.968416    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-kube-api-access-8x4kc\" (UniqueName: \"kubernetes.io/projected/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-kube-api-access-8x4kc\") pod \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\" (UID: \"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e\") "
Oct 30 19:14:03.972788 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.969420    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:03.972788 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.970999    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-maas-api-tls" pod "97e8b1ea-88f1-4f81-80b1-2b7c81e6926e" (UID: "97e8b1ea-88f1-4f81-80b1-2b7c81e6926e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:03.974230 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.974207    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-metrics-certs" pod "97e8b1ea-88f1-4f81-80b1-2b7c81e6926e" (UID: "97e8b1ea-88f1-4f81-80b1-2b7c81e6926e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:03.975080 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:03.974926    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-kube-api-access-8x4kc" pod "97e8b1ea-88f1-4f81-80b1-2b7c81e6926e" (UID: "97e8b1ea-88f1-4f81-80b1-2b7c81e6926e"). InnerVolumeSpecName "kube-api-access-8x4kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:04.070999 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:04.070959    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x4kc\" (UniqueName: \"kubernetes.io/projected/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-kube-api-access-8x4kc\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:04.070999 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:04.070999    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:04.071278 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:04.071012    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:04.680166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:04.673417    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b" event={"ID":"97e8b1ea-88f1-4f81-80b1-2b7c81e6926e","Type":"ContainerDied","Data":"ac5a639a269a26f813dc690a6f28782f9feec279a1b398f3290080a6052e0b8a"}
Oct 30 19:14:04.680166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:04.673489    6014 scope.go:122] "RemoveContainer" containerID="fd1d8f3b98dd3b3e080f48ecec733594c66f6727be5ca881166b10309dc988f0"
Oct 30 19:14:04.680166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:04.673654    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"
Oct 30 19:14:04.838706 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:04.838670    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"]
Oct 30 19:14:04.888173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:04.888125    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2e004a-7f4b7cf547-9bl6b"]
Oct 30 19:14:05.479859 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:05.479822    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e8b1ea-88f1-4f81-80b1-2b7c81e6926e" path="/var/lib/kubelet/pods/97e8b1ea-88f1-4f81-80b1-2b7c81e6926e/volumes"
Oct 30 19:14:12.441725 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:12.441686    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"]
Oct 30 19:14:12.442231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:12.441948    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk" podUID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerName="istio-proxy" containerID="cri-o://d892fd66ebd609b79c6bd7c5ef6c1d9e1dda24ff5ed58542b798151f24929cc2" gracePeriod=30
Oct 30 19:14:17.728984 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.728957    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:14:17.781228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.781132    6014 generic.go:356] "Generic (PLEG): container finished" podID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerID="d892fd66ebd609b79c6bd7c5ef6c1d9e1dda24ff5ed58542b798151f24929cc2" exitCode=0
Oct 30 19:14:17.781516 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.781236    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"
Oct 30 19:14:17.781618 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.781257    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk" event={"ID":"f081490d-ac76-40cc-9df9-14aafe2726ad","Type":"ContainerDied","Data":"d892fd66ebd609b79c6bd7c5ef6c1d9e1dda24ff5ed58542b798151f24929cc2"}
Oct 30 19:14:17.781666 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.781626    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk" event={"ID":"f081490d-ac76-40cc-9df9-14aafe2726ad","Type":"ContainerDied","Data":"b0c159af3c533ff753998c6a76fa4f1d468f9698c895f2ed05deb0c454816583"}
Oct 30 19:14:17.781666 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.781655    6014 scope.go:122] "RemoveContainer" containerID="d892fd66ebd609b79c6bd7c5ef6c1d9e1dda24ff5ed58542b798151f24929cc2"
Oct 30 19:14:17.793246 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.793223    6014 scope.go:122] "RemoveContainer" containerID="cd629341e9b682c67cb34fdec95952f1a47018a6dd68c251c594f7e6ba365728"
Oct 30 19:14:17.806503 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.806479    6014 scope.go:122] "RemoveContainer" containerID="d892fd66ebd609b79c6bd7c5ef6c1d9e1dda24ff5ed58542b798151f24929cc2"
Oct 30 19:14:17.806826 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:14:17.806803    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d892fd66ebd609b79c6bd7c5ef6c1d9e1dda24ff5ed58542b798151f24929cc2\": container with ID starting with d892fd66ebd609b79c6bd7c5ef6c1d9e1dda24ff5ed58542b798151f24929cc2 not found: ID does not exist" containerID="d892fd66ebd609b79c6bd7c5ef6c1d9e1dda24ff5ed58542b798151f24929cc2"
Oct 30 19:14:17.806892 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.806840    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d892fd66ebd609b79c6bd7c5ef6c1d9e1dda24ff5ed58542b798151f24929cc2"} err="failed to get container status \"d892fd66ebd609b79c6bd7c5ef6c1d9e1dda24ff5ed58542b798151f24929cc2\": rpc error: code = NotFound desc = could not find container \"d892fd66ebd609b79c6bd7c5ef6c1d9e1dda24ff5ed58542b798151f24929cc2\": container with ID starting with d892fd66ebd609b79c6bd7c5ef6c1d9e1dda24ff5ed58542b798151f24929cc2 not found: ID does not exist"
Oct 30 19:14:17.806892 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.806867    6014 scope.go:122] "RemoveContainer" containerID="cd629341e9b682c67cb34fdec95952f1a47018a6dd68c251c594f7e6ba365728"
Oct 30 19:14:17.807269 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:14:17.807241    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd629341e9b682c67cb34fdec95952f1a47018a6dd68c251c594f7e6ba365728\": container with ID starting with cd629341e9b682c67cb34fdec95952f1a47018a6dd68c251c594f7e6ba365728 not found: ID does not exist" containerID="cd629341e9b682c67cb34fdec95952f1a47018a6dd68c251c594f7e6ba365728"
Oct 30 19:14:17.807331 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.807268    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd629341e9b682c67cb34fdec95952f1a47018a6dd68c251c594f7e6ba365728"} err="failed to get container status \"cd629341e9b682c67cb34fdec95952f1a47018a6dd68c251c594f7e6ba365728\": rpc error: code = NotFound desc = could not find container \"cd629341e9b682c67cb34fdec95952f1a47018a6dd68c251c594f7e6ba365728\": container with ID starting with cd629341e9b682c67cb34fdec95952f1a47018a6dd68c251c594f7e6ba365728 not found: ID does not exist"
Oct 30 19:14:17.834970 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.834940    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f081490d-ac76-40cc-9df9-14aafe2726ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f081490d-ac76-40cc-9df9-14aafe2726ad-istiod-ca-cert\") pod \"f081490d-ac76-40cc-9df9-14aafe2726ad\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") "
Oct 30 19:14:17.835173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.835011    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-podinfo\") pod \"f081490d-ac76-40cc-9df9-14aafe2726ad\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") "
Oct 30 19:14:17.835173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.835042    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-data\") pod \"f081490d-ac76-40cc-9df9-14aafe2726ad\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") "
Oct 30 19:14:17.835173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.835092    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-token\" (UniqueName: \"kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-token\") pod \"f081490d-ac76-40cc-9df9-14aafe2726ad\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") "
Oct 30 19:14:17.835173 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.835131    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-envoy\") pod \"f081490d-ac76-40cc-9df9-14aafe2726ad\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") "
Oct 30 19:14:17.835409 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.835235    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-socket\") pod \"f081490d-ac76-40cc-9df9-14aafe2726ad\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") "
Oct 30 19:14:17.835409 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.835287    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-kube-api-access-nd595\" (UniqueName: \"kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-kube-api-access-nd595\") pod \"f081490d-ac76-40cc-9df9-14aafe2726ad\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") "
Oct 30 19:14:17.835409 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.835342    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-certs\") pod \"f081490d-ac76-40cc-9df9-14aafe2726ad\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") "
Oct 30 19:14:17.835575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.835406    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-data" pod "f081490d-ac76-40cc-9df9-14aafe2726ad" (UID: "f081490d-ac76-40cc-9df9-14aafe2726ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:17.835575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.835438    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f081490d-ac76-40cc-9df9-14aafe2726ad-istiod-ca-cert" pod "f081490d-ac76-40cc-9df9-14aafe2726ad" (UID: "f081490d-ac76-40cc-9df9-14aafe2726ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:17.835575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.835553    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-socket" pod "f081490d-ac76-40cc-9df9-14aafe2726ad" (UID: "f081490d-ac76-40cc-9df9-14aafe2726ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:17.835746 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.835598    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-credential-socket\") pod \"f081490d-ac76-40cc-9df9-14aafe2726ad\" (UID: \"f081490d-ac76-40cc-9df9-14aafe2726ad\") "
Oct 30 19:14:17.835809 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.835783    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-certs" pod "f081490d-ac76-40cc-9df9-14aafe2726ad" (UID: "f081490d-ac76-40cc-9df9-14aafe2726ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:17.836069 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.836046    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-credential-socket" pod "f081490d-ac76-40cc-9df9-14aafe2726ad" (UID: "f081490d-ac76-40cc-9df9-14aafe2726ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:17.836522 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.836501    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:17.836581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.836531    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f081490d-ac76-40cc-9df9-14aafe2726ad-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:17.836581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.836546    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:17.836581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.836561    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:17.836581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.836574    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:17.837580 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.837561    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-token" pod "f081490d-ac76-40cc-9df9-14aafe2726ad" (UID: "f081490d-ac76-40cc-9df9-14aafe2726ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:17.837670 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.837581    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-podinfo" pod "f081490d-ac76-40cc-9df9-14aafe2726ad" (UID: "f081490d-ac76-40cc-9df9-14aafe2726ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:17.838021 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.838000    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-envoy" pod "f081490d-ac76-40cc-9df9-14aafe2726ad" (UID: "f081490d-ac76-40cc-9df9-14aafe2726ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:17.838021 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.838008    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-kube-api-access-nd595" pod "f081490d-ac76-40cc-9df9-14aafe2726ad" (UID: "f081490d-ac76-40cc-9df9-14aafe2726ad"). InnerVolumeSpecName "kube-api-access-nd595". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:17.938041 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.938005    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:17.938041 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.938038    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:17.938041 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.938048    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f081490d-ac76-40cc-9df9-14aafe2726ad-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:17.938320 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:17.938058    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd595\" (UniqueName: \"kubernetes.io/projected/f081490d-ac76-40cc-9df9-14aafe2726ad-kube-api-access-nd595\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:18.203900 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:18.203866    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"]
Oct 30 19:14:18.207163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:18.207111    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2e004a-openshift-default-66b9864bdc-sfqpk"]
Oct 30 19:14:19.478638 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:19.478602    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f081490d-ac76-40cc-9df9-14aafe2726ad" path="/var/lib/kubelet/pods/f081490d-ac76-40cc-9df9-14aafe2726ad/volumes"
Oct 30 19:14:19.949723 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:19.949691    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"]
Oct 30 19:14:19.949923 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:19.949902    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6" podUID="1dfe5333-8253-43be-b7fe-c548ff82067c" containerName="istio-proxy" containerID="cri-o://77e639fe12a0ff3936ade9bfea43b08746d3822bd1033f80091dd4ee2d68eed6" gracePeriod=30
Oct 30 19:14:20.451092 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.451054    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"]
Oct 30 19:14:20.451817 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.451799    6014 cpu_manager.go:414] "containerMap: removing container" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerName="istio-proxy"
Oct 30 19:14:20.451869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.451818    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerName="istio-proxy"
Oct 30 19:14:20.451869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.451833    6014 cpu_manager.go:414] "containerMap: removing container" podUID="577ef8da-25fd-49ff-bbd4-dc8e568fc99b" containerName="payload-processing"
Oct 30 19:14:20.451869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.451839    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577ef8da-25fd-49ff-bbd4-dc8e568fc99b" containerName="payload-processing"
Oct 30 19:14:20.451869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.451850    6014 cpu_manager.go:414] "containerMap: removing container" podUID="97e8b1ea-88f1-4f81-80b1-2b7c81e6926e" containerName="maas-api"
Oct 30 19:14:20.451869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.451857    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e8b1ea-88f1-4f81-80b1-2b7c81e6926e" containerName="maas-api"
Oct 30 19:14:20.452036 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.451896    6014 cpu_manager.go:414] "containerMap: removing container" podUID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerName="istio-proxy"
Oct 30 19:14:20.452036 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.451901    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerName="istio-proxy"
Oct 30 19:14:20.452036 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.451922    6014 cpu_manager.go:414] "containerMap: removing container" podUID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerName="istio-proxy"
Oct 30 19:14:20.452036 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.451927    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerName="istio-proxy"
Oct 30 19:14:20.452036 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.452021    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerName="istio-proxy"
Oct 30 19:14:20.452036 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.452033    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerName="istio-proxy"
Oct 30 19:14:20.452243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.452039    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerName="istio-proxy"
Oct 30 19:14:20.452243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.452047    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="577ef8da-25fd-49ff-bbd4-dc8e568fc99b" containerName="payload-processing"
Oct 30 19:14:20.452243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.452053    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e8b1ea-88f1-4f81-80b1-2b7c81e6926e" containerName="maas-api"
Oct 30 19:14:20.457287 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.457267    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:20.509886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.509844    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8da6e2-serving-cert\""
Oct 30 19:14:20.513519 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.513485    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"]
Oct 30 19:14:20.577049 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.577020    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2k4k\" (UniqueName: \"kubernetes.io/projected/7301bcc9-644f-4488-aa20-6405d959b866-kube-api-access-p2k4k\") pod \"maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:20.577257 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.577132    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7301bcc9-644f-4488-aa20-6405d959b866-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:20.577421 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.577400    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-metrics-certs\") pod \"maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:20.577505 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.577488    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-maas-api-tls\") pod \"maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:20.678764 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.678725    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-metrics-certs\") pod \"maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:20.678966 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.678818    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-maas-api-tls\") pod \"maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:20.678966 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.678912    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2k4k\" (UniqueName: \"kubernetes.io/projected/7301bcc9-644f-4488-aa20-6405d959b866-kube-api-access-p2k4k\") pod \"maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:20.679093 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:14:20.678978    6014 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-serving-cert: secret "maas-api-e2e-inf-a-8da6e2-serving-cert" not found
Oct 30 19:14:20.679093 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.678981    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7301bcc9-644f-4488-aa20-6405d959b866-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:20.679093 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:14:20.679058    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-maas-api-tls podName:7301bcc9-644f-4488-aa20-6405d959b866 nodeName:}" failed. No retries permitted until 2026-10-30 19:14:21.179035154 +0000 UTC m=+2099.822202767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-maas-api-tls") pod "maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp" (UID: "7301bcc9-644f-4488-aa20-6405d959b866") : secret "maas-api-e2e-inf-a-8da6e2-serving-cert" not found
Oct 30 19:14:20.679693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.679672    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7301bcc9-644f-4488-aa20-6405d959b866-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:20.681471 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.681451    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-metrics-certs\") pod \"maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:20.737089 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:20.737013    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2k4k\" (UniqueName: \"kubernetes.io/projected/7301bcc9-644f-4488-aa20-6405d959b866-kube-api-access-p2k4k\") pod \"maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:21.187180 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:21.187103    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-maas-api-tls\") pod \"maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:21.189815 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:21.189791    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-maas-api-tls\") pod \"maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:21.375693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:21.375656    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:21.709072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:21.708985    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:14:21.757180 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:21.757115    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"]
Oct 30 19:14:21.758319 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:14:21.758278    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7301bcc9_644f_4488_aa20_6405d959b866.slice/crio-1c993da102ead301971910505d90d76e02a69bc61a52c667ac2d34b4784af2c4 WatchSource:0}: Error finding container 1c993da102ead301971910505d90d76e02a69bc61a52c667ac2d34b4784af2c4: Status 404 returned error can't find the container with id 1c993da102ead301971910505d90d76e02a69bc61a52c667ac2d34b4784af2c4
Oct 30 19:14:21.760267 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:21.760246    6014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:14:21.814461 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:21.814424    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp" event={"ID":"7301bcc9-644f-4488-aa20-6405d959b866","Type":"ContainerStarted","Data":"1c993da102ead301971910505d90d76e02a69bc61a52c667ac2d34b4784af2c4"}
Oct 30 19:14:22.825113 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:22.825012    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp" event={"ID":"7301bcc9-644f-4488-aa20-6405d959b866","Type":"ContainerStarted","Data":"87c960c01c1c0dec605ff7bfb79b81bbcb021cf0bfb37320fd4844da53f2842b"}
Oct 30 19:14:22.825608 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:22.825204    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:22.945336 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:22.945287    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp" podStartSLOduration=2.135740612 podStartE2EDuration="2.94527027s" totalImagesPullingTime="809.529658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:22.941222682 +0000 UTC m=+2101.584390295" watchObservedRunningTime="2026-10-30 19:14:22.94527027 +0000 UTC m=+2101.588437885"
Oct 30 19:14:23.263203 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:14:23.263109    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:14:24.403974 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:24.403943    6014 scope.go:122] "RemoveContainer" containerID="90f17c40a12c2696c43c6b9301efe888320cc33c946b98eb5aa5b901be9e0524"
Oct 30 19:14:25.030204 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.030160    6014 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:14:25.030360 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.030233    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6" podUID="1dfe5333-8253-43be-b7fe-c548ff82067c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 19:14:25.221599 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.221527    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:14:25.354265 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.354236    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-token\" (UniqueName: \"kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-token\") pod \"1dfe5333-8253-43be-b7fe-c548ff82067c\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") "
Oct 30 19:14:25.354464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.354271    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-podinfo\") pod \"1dfe5333-8253-43be-b7fe-c548ff82067c\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") "
Oct 30 19:14:25.354464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.354406    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-envoy\") pod \"1dfe5333-8253-43be-b7fe-c548ff82067c\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") "
Oct 30 19:14:25.354571 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.354497    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-credential-socket\") pod \"1dfe5333-8253-43be-b7fe-c548ff82067c\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") "
Oct 30 19:14:25.354571 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.354563    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-certs\") pod \"1dfe5333-8253-43be-b7fe-c548ff82067c\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") "
Oct 30 19:14:25.354735 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.354717    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-data\") pod \"1dfe5333-8253-43be-b7fe-c548ff82067c\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") "
Oct 30 19:14:25.354816 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.354775    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-socket\") pod \"1dfe5333-8253-43be-b7fe-c548ff82067c\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") "
Oct 30 19:14:25.354816 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.354795    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-credential-socket" pod "1dfe5333-8253-43be-b7fe-c548ff82067c" (UID: "1dfe5333-8253-43be-b7fe-c548ff82067c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:25.354937 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.354873    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-kube-api-access-xnx2n\" (UniqueName: \"kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-kube-api-access-xnx2n\") pod \"1dfe5333-8253-43be-b7fe-c548ff82067c\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") "
Oct 30 19:14:25.354992 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.354937    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dfe5333-8253-43be-b7fe-c548ff82067c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dfe5333-8253-43be-b7fe-c548ff82067c-istiod-ca-cert\") pod \"1dfe5333-8253-43be-b7fe-c548ff82067c\" (UID: \"1dfe5333-8253-43be-b7fe-c548ff82067c\") "
Oct 30 19:14:25.355045 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.355017    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-socket" pod "1dfe5333-8253-43be-b7fe-c548ff82067c" (UID: "1dfe5333-8253-43be-b7fe-c548ff82067c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:25.355610 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.355274    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-certs" pod "1dfe5333-8253-43be-b7fe-c548ff82067c" (UID: "1dfe5333-8253-43be-b7fe-c548ff82067c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:25.355610 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.355309    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-data" pod "1dfe5333-8253-43be-b7fe-c548ff82067c" (UID: "1dfe5333-8253-43be-b7fe-c548ff82067c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:25.355610 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.355578    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfe5333-8253-43be-b7fe-c548ff82067c-istiod-ca-cert" pod "1dfe5333-8253-43be-b7fe-c548ff82067c" (UID: "1dfe5333-8253-43be-b7fe-c548ff82067c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:25.355957 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.355933    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.356030 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.355964    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.356030 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.355980    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.356030 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.355994    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.356030 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.356008    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dfe5333-8253-43be-b7fe-c548ff82067c-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.357774 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.357748    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-podinfo" pod "1dfe5333-8253-43be-b7fe-c548ff82067c" (UID: "1dfe5333-8253-43be-b7fe-c548ff82067c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:25.357892 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.357797    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-envoy" pod "1dfe5333-8253-43be-b7fe-c548ff82067c" (UID: "1dfe5333-8253-43be-b7fe-c548ff82067c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:25.357892 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.357799    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-token" pod "1dfe5333-8253-43be-b7fe-c548ff82067c" (UID: "1dfe5333-8253-43be-b7fe-c548ff82067c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:25.358579 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.358552    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-kube-api-access-xnx2n" pod "1dfe5333-8253-43be-b7fe-c548ff82067c" (UID: "1dfe5333-8253-43be-b7fe-c548ff82067c"). InnerVolumeSpecName "kube-api-access-xnx2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:25.457847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.457813    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnx2n\" (UniqueName: \"kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-kube-api-access-xnx2n\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.457847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.457845    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.457847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.457855    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.458317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.457863    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dfe5333-8253-43be-b7fe-c548ff82067c-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.860425 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.860393    6014 generic.go:356] "Generic (PLEG): container finished" podID="1dfe5333-8253-43be-b7fe-c548ff82067c" containerID="77e639fe12a0ff3936ade9bfea43b08746d3822bd1033f80091dd4ee2d68eed6" exitCode=0
Oct 30 19:14:25.860581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.860441    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"
Oct 30 19:14:25.860581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.860478    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6" event={"ID":"1dfe5333-8253-43be-b7fe-c548ff82067c","Type":"ContainerDied","Data":"77e639fe12a0ff3936ade9bfea43b08746d3822bd1033f80091dd4ee2d68eed6"}
Oct 30 19:14:25.860581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.860536    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6" event={"ID":"1dfe5333-8253-43be-b7fe-c548ff82067c","Type":"ContainerDied","Data":"38ede00d5c90a85aeaddc53184e298f2a984dd4a6dc079cafaef73cff3010975"}
Oct 30 19:14:25.860581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.860562    6014 scope.go:122] "RemoveContainer" containerID="77e639fe12a0ff3936ade9bfea43b08746d3822bd1033f80091dd4ee2d68eed6"
Oct 30 19:14:25.872404 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.872384    6014 scope.go:122] "RemoveContainer" containerID="77e639fe12a0ff3936ade9bfea43b08746d3822bd1033f80091dd4ee2d68eed6"
Oct 30 19:14:25.872717 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:14:25.872695    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e639fe12a0ff3936ade9bfea43b08746d3822bd1033f80091dd4ee2d68eed6\": container with ID starting with 77e639fe12a0ff3936ade9bfea43b08746d3822bd1033f80091dd4ee2d68eed6 not found: ID does not exist" containerID="77e639fe12a0ff3936ade9bfea43b08746d3822bd1033f80091dd4ee2d68eed6"
Oct 30 19:14:25.872810 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:25.872726    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e639fe12a0ff3936ade9bfea43b08746d3822bd1033f80091dd4ee2d68eed6"} err="failed to get container status \"77e639fe12a0ff3936ade9bfea43b08746d3822bd1033f80091dd4ee2d68eed6\": rpc error: code = NotFound desc = could not find container \"77e639fe12a0ff3936ade9bfea43b08746d3822bd1033f80091dd4ee2d68eed6\": container with ID starting with 77e639fe12a0ff3936ade9bfea43b08746d3822bd1033f80091dd4ee2d68eed6 not found: ID does not exist"
Oct 30 19:14:26.024747 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:26.024717    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"]
Oct 30 19:14:26.073552 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:26.073516    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-57rw6"]
Oct 30 19:14:27.478538 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:27.478501    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dfe5333-8253-43be-b7fe-c548ff82067c" path="/var/lib/kubelet/pods/1dfe5333-8253-43be-b7fe-c548ff82067c/volumes"
Oct 30 19:14:28.850721 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:28.850691    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:14:53.259942 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:14:53.259906    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:14:56.890923 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:56.890887    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"]
Oct 30 19:14:56.891722 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:56.891704    6014 cpu_manager.go:414] "containerMap: removing container" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerName="istio-proxy"
Oct 30 19:14:56.891770 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:56.891724    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a2f9fa-4621-45b8-84a8-2aa9721df791" containerName="istio-proxy"
Oct 30 19:14:56.891770 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:56.891740    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1dfe5333-8253-43be-b7fe-c548ff82067c" containerName="istio-proxy"
Oct 30 19:14:56.891770 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:56.891747    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfe5333-8253-43be-b7fe-c548ff82067c" containerName="istio-proxy"
Oct 30 19:14:56.891878 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:56.891865    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f081490d-ac76-40cc-9df9-14aafe2726ad" containerName="istio-proxy"
Oct 30 19:14:56.891917 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:56.891890    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfe5333-8253-43be-b7fe-c548ff82067c" containerName="istio-proxy"
Oct 30 19:14:56.895488 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:56.895468    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:56.947178 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:56.947101    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e0fc1f-serving-cert\""
Oct 30 19:14:56.964130 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:56.964095    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-metrics-certs\") pod \"maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:56.964352 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:56.964237    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr5b8\" (UniqueName: \"kubernetes.io/projected/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-kube-api-access-tr5b8\") pod \"maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:56.964352 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:56.964309    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:56.964471 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:56.964401    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:57.002241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:57.002195    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"]
Oct 30 19:14:57.065829 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:57.065794    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-metrics-certs\") pod \"maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:57.066008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:57.065849    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr5b8\" (UniqueName: \"kubernetes.io/projected/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-kube-api-access-tr5b8\") pod \"maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:57.066008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:57.065888    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:57.066008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:57.065955    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:57.066207 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:14:57.066182    6014 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-serving-cert: secret "maas-api-e2e-inf-a-e0fc1f-serving-cert" not found
Oct 30 19:14:57.066295 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:14:57.066283    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-maas-api-tls podName:a99523a9-e39a-4c4d-b06d-9e27c847ffd5 nodeName:}" failed. No retries permitted until 2026-10-30 19:14:57.566260176 +0000 UTC m=+2136.209427784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-maas-api-tls") pod "maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk" (UID: "a99523a9-e39a-4c4d-b06d-9e27c847ffd5") : secret "maas-api-e2e-inf-a-e0fc1f-serving-cert" not found
Oct 30 19:14:57.066632 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:57.066614    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:57.068585 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:57.068562    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-metrics-certs\") pod \"maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:57.125427 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:57.125390    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr5b8\" (UniqueName: \"kubernetes.io/projected/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-kube-api-access-tr5b8\") pod \"maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:57.579896 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:57.579853    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:57.583182 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:57.583118    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:57.809687 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:57.809655    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:14:58.193621 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:58.193596    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"]
Oct 30 19:14:58.194782 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:14:58.194756    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99523a9_e39a_4c4d_b06d_9e27c847ffd5.slice/crio-470842a9bc9ef07b509581a2ee70bb6ad602e8f8ff5533bfb231ee3776fb50b2 WatchSource:0}: Error finding container 470842a9bc9ef07b509581a2ee70bb6ad602e8f8ff5533bfb231ee3776fb50b2: Status 404 returned error can't find the container with id 470842a9bc9ef07b509581a2ee70bb6ad602e8f8ff5533bfb231ee3776fb50b2
Oct 30 19:14:59.112570 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:14:59.112515    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk" event={"ID":"a99523a9-e39a-4c4d-b06d-9e27c847ffd5","Type":"ContainerStarted","Data":"470842a9bc9ef07b509581a2ee70bb6ad602e8f8ff5533bfb231ee3776fb50b2"}
Oct 30 19:15:00.121665 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.121630    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk" event={"ID":"a99523a9-e39a-4c4d-b06d-9e27c847ffd5","Type":"ContainerStarted","Data":"d6347e4aec71cf935a5506c0dbc0bd5518f37de8c7e8e0084f7f63fa3feb281e"}
Oct 30 19:15:00.122227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.121819    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:15:00.210587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.210549    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"]
Oct 30 19:15:00.215215 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.215193    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"]
Oct 30 19:15:00.215398 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.215306    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"
Oct 30 19:15:00.220325 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.220300    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"
Oct 30 19:15:00.240083 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.240058    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"]
Oct 30 19:15:00.244648 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.244628    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"
Oct 30 19:15:00.245627 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.245611    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"]
Oct 30 19:15:00.250105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.250084    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"
Oct 30 19:15:00.270519 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.270493    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"]
Oct 30 19:15:00.273508 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.273486    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"]
Oct 30 19:15:00.276700 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.276660    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk" podStartSLOduration=3.383708709 podStartE2EDuration="4.276648625s" totalImagesPullingTime="892.939916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:00.27549543 +0000 UTC m=+2138.918663045" watchObservedRunningTime="2026-10-30 19:15:00.276648625 +0000 UTC m=+2138.919816238"
Oct 30 19:15:00.278165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.278124    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"
Oct 30 19:15:00.317767 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.317727    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"]
Oct 30 19:15:00.320475 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.320448    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4f0028-19de-42ed-a16a-113f75252fc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h\" (UID: \"1a4f0028-19de-42ed-a16a-113f75252fc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"
Oct 30 19:15:00.320610 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.320584    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz\" (UID: \"8f930fa4-482b-45e8-be8a-42bcbc4fa86f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"
Oct 30 19:15:00.320656 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.320632    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8bdh\" (UniqueName: \"kubernetes.io/projected/a9df751a-8363-49aa-ac14-e37ab07d6273-kube-api-access-t8bdh\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg\" (UID: \"a9df751a-8363-49aa-ac14-e37ab07d6273\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"
Oct 30 19:15:00.320720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.320675    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9df751a-8363-49aa-ac14-e37ab07d6273-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg\" (UID: \"a9df751a-8363-49aa-ac14-e37ab07d6273\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"
Oct 30 19:15:00.320768 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.320728    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zfmx\" (UniqueName: \"kubernetes.io/projected/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-kube-api-access-7zfmx\") pod \"maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz\" (UID: \"8f930fa4-482b-45e8-be8a-42bcbc4fa86f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"
Oct 30 19:15:00.320812 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.320763    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-rpljm\" (UID: \"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"
Oct 30 19:15:00.320847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.320839    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chcr5\" (UniqueName: \"kubernetes.io/projected/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-kube-api-access-chcr5\") pod \"maas-api-key-cleanup-29889795-rpljm\" (UID: \"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"
Oct 30 19:15:00.320981 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.320963    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pltz4\" (UniqueName: \"kubernetes.io/projected/1a4f0028-19de-42ed-a16a-113f75252fc4-kube-api-access-pltz4\") pod \"maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h\" (UID: \"1a4f0028-19de-42ed-a16a-113f75252fc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"
Oct 30 19:15:00.329402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.329368    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:15:00.380513 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.380214    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"]
Oct 30 19:15:00.387156 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.386558    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"]
Oct 30 19:15:00.390215 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.390178    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"]
Oct 30 19:15:00.443167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.439242    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4f0028-19de-42ed-a16a-113f75252fc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h\" (UID: \"1a4f0028-19de-42ed-a16a-113f75252fc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"
Oct 30 19:15:00.443167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.439404    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz\" (UID: \"8f930fa4-482b-45e8-be8a-42bcbc4fa86f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"
Oct 30 19:15:00.443167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.439432    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8bdh\" (UniqueName: \"kubernetes.io/projected/a9df751a-8363-49aa-ac14-e37ab07d6273-kube-api-access-t8bdh\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg\" (UID: \"a9df751a-8363-49aa-ac14-e37ab07d6273\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"
Oct 30 19:15:00.443167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.439479    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9df751a-8363-49aa-ac14-e37ab07d6273-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg\" (UID: \"a9df751a-8363-49aa-ac14-e37ab07d6273\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"
Oct 30 19:15:00.443167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.439557    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zfmx\" (UniqueName: \"kubernetes.io/projected/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-kube-api-access-7zfmx\") pod \"maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz\" (UID: \"8f930fa4-482b-45e8-be8a-42bcbc4fa86f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"
Oct 30 19:15:00.443167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.439599    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-rpljm\" (UID: \"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"
Oct 30 19:15:00.443167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.439680    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chcr5\" (UniqueName: \"kubernetes.io/projected/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-kube-api-access-chcr5\") pod \"maas-api-key-cleanup-29889795-rpljm\" (UID: \"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"
Oct 30 19:15:00.443167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.439828    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pltz4\" (UniqueName: \"kubernetes.io/projected/1a4f0028-19de-42ed-a16a-113f75252fc4-kube-api-access-pltz4\") pod \"maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h\" (UID: \"1a4f0028-19de-42ed-a16a-113f75252fc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"
Oct 30 19:15:00.443167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.439884    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6\" (UID: \"b18e5c80-93bc-41a1-902a-d5f159e5c1c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"
Oct 30 19:15:00.443167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.439914    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlpcr\" (UniqueName: \"kubernetes.io/projected/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-kube-api-access-vlpcr\") pod \"maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6\" (UID: \"b18e5c80-93bc-41a1-902a-d5f159e5c1c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"
Oct 30 19:15:00.443167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.442241    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4f0028-19de-42ed-a16a-113f75252fc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h\" (UID: \"1a4f0028-19de-42ed-a16a-113f75252fc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"
Oct 30 19:15:00.443167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.442876    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-rpljm\" (UID: \"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"
Oct 30 19:15:00.443946 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.443424    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9df751a-8363-49aa-ac14-e37ab07d6273-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg\" (UID: \"a9df751a-8363-49aa-ac14-e37ab07d6273\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"
Oct 30 19:15:00.443946 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.443600    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz\" (UID: \"8f930fa4-482b-45e8-be8a-42bcbc4fa86f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"
Oct 30 19:15:00.515161 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.511404    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8bdh\" (UniqueName: \"kubernetes.io/projected/a9df751a-8363-49aa-ac14-e37ab07d6273-kube-api-access-t8bdh\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg\" (UID: \"a9df751a-8363-49aa-ac14-e37ab07d6273\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"
Oct 30 19:15:00.525278 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.516884    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zfmx\" (UniqueName: \"kubernetes.io/projected/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-kube-api-access-7zfmx\") pod \"maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz\" (UID: \"8f930fa4-482b-45e8-be8a-42bcbc4fa86f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"
Oct 30 19:15:00.525278 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.517790    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chcr5\" (UniqueName: \"kubernetes.io/projected/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-kube-api-access-chcr5\") pod \"maas-api-key-cleanup-29889795-rpljm\" (UID: \"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"
Oct 30 19:15:00.525278 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.523974    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pltz4\" (UniqueName: \"kubernetes.io/projected/1a4f0028-19de-42ed-a16a-113f75252fc4-kube-api-access-pltz4\") pod \"maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h\" (UID: \"1a4f0028-19de-42ed-a16a-113f75252fc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"
Oct 30 19:15:00.531378 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.531348    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"
Oct 30 19:15:00.538484 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.538460    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"
Oct 30 19:15:00.541268 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.541245    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6\" (UID: \"b18e5c80-93bc-41a1-902a-d5f159e5c1c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"
Oct 30 19:15:00.541390 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.541282    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpcr\" (UniqueName: \"kubernetes.io/projected/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-kube-api-access-vlpcr\") pod \"maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6\" (UID: \"b18e5c80-93bc-41a1-902a-d5f159e5c1c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"
Oct 30 19:15:00.542309 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.542289    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6\" (UID: \"b18e5c80-93bc-41a1-902a-d5f159e5c1c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"
Oct 30 19:15:00.562555 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.562526    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"
Oct 30 19:15:00.569298 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.568898    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"
Oct 30 19:15:00.601404 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.601337    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpcr\" (UniqueName: \"kubernetes.io/projected/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-kube-api-access-vlpcr\") pod \"maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6\" (UID: \"b18e5c80-93bc-41a1-902a-d5f159e5c1c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"
Oct 30 19:15:00.892703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.892672    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"
Oct 30 19:15:00.932351 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.932326    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"]
Oct 30 19:15:00.934832 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:15:00.934807    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf54a96_1b68_4a93_b3d7_f6cbdc7d76af.slice/crio-41edcb09a9c286bc6f01cd76d278ff031e1cedf0a035cb10f7855dd219646479 WatchSource:0}: Error finding container 41edcb09a9c286bc6f01cd76d278ff031e1cedf0a035cb10f7855dd219646479: Status 404 returned error can't find the container with id 41edcb09a9c286bc6f01cd76d278ff031e1cedf0a035cb10f7855dd219646479
Oct 30 19:15:00.987011 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:00.986979    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"]
Oct 30 19:15:00.989001 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:15:00.988969    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4f0028_19de_42ed_a16a_113f75252fc4.slice/crio-e19c40959293fafccb7f6fcc6f538b19ef5c2dcd9147c64cb4a4c369ab566b30 WatchSource:0}: Error finding container e19c40959293fafccb7f6fcc6f538b19ef5c2dcd9147c64cb4a4c369ab566b30: Status 404 returned error can't find the container with id e19c40959293fafccb7f6fcc6f538b19ef5c2dcd9147c64cb4a4c369ab566b30
Oct 30 19:15:01.040441 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:01.037332    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"]
Oct 30 19:15:01.044036 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:01.042927    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"]
Oct 30 19:15:01.129673 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:01.129637    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm" event={"ID":"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af","Type":"ContainerStarted","Data":"41edcb09a9c286bc6f01cd76d278ff031e1cedf0a035cb10f7855dd219646479"}
Oct 30 19:15:01.130990 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:01.130955    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h" event={"ID":"1a4f0028-19de-42ed-a16a-113f75252fc4","Type":"ContainerStarted","Data":"e19c40959293fafccb7f6fcc6f538b19ef5c2dcd9147c64cb4a4c369ab566b30"}
Oct 30 19:15:01.132550 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:01.132488    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz" event={"ID":"8f930fa4-482b-45e8-be8a-42bcbc4fa86f","Type":"ContainerStarted","Data":"5409a28862409ee14593ce3f30fcab830e3f4889fe2bbed1df3905f12ce394cf"}
Oct 30 19:15:01.133790 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:01.133770    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg" event={"ID":"a9df751a-8363-49aa-ac14-e37ab07d6273","Type":"ContainerStarted","Data":"7911462025a08f8bcb511f7974be945056675625784d16c76a746a3dbf8dcd33"}
Oct 30 19:15:01.277018 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:01.276987    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"]
Oct 30 19:15:01.278333 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:15:01.278306    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18e5c80_93bc_41a1_902a_d5f159e5c1c7.slice/crio-479c3b4adff50917700cf9c2fbe982528c946520dab1cc5aa003d58de738e21c WatchSource:0}: Error finding container 479c3b4adff50917700cf9c2fbe982528c946520dab1cc5aa003d58de738e21c: Status 404 returned error can't find the container with id 479c3b4adff50917700cf9c2fbe982528c946520dab1cc5aa003d58de738e21c
Oct 30 19:15:02.143792 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:02.143755    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6" event={"ID":"b18e5c80-93bc-41a1-902a-d5f159e5c1c7","Type":"ContainerStarted","Data":"da9167484c52ea62e2918821a20c97217cdb296632a1e0c3f7a2b4b94c04ceb4"}
Oct 30 19:15:02.144324 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:02.143817    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6" event={"ID":"b18e5c80-93bc-41a1-902a-d5f159e5c1c7","Type":"ContainerStarted","Data":"479c3b4adff50917700cf9c2fbe982528c946520dab1cc5aa003d58de738e21c"}
Oct 30 19:15:02.145457 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:02.145428    6014 generic.go:356] "Generic (PLEG): container finished" podID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" containerID="0c91e2d8cd575f4ff158a41450775c4d582cb11a11009f08d3d42657d54b88d3" exitCode=60
Oct 30 19:15:02.145606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:02.145460    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm" event={"ID":"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af","Type":"ContainerDied","Data":"0c91e2d8cd575f4ff158a41450775c4d582cb11a11009f08d3d42657d54b88d3"}
Oct 30 19:15:02.145802 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:02.145782    6014 scope.go:122] "RemoveContainer" containerID="0c91e2d8cd575f4ff158a41450775c4d582cb11a11009f08d3d42657d54b88d3"
Oct 30 19:15:02.147362 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:02.147340    6014 generic.go:356] "Generic (PLEG): container finished" podID="1a4f0028-19de-42ed-a16a-113f75252fc4" containerID="9ee3de9d17a525dd1cd4e59abf08602d89d5a22ed3b1c2e58a02980b61fbfbc1" exitCode=60
Oct 30 19:15:02.147457 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:02.147441    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h" event={"ID":"1a4f0028-19de-42ed-a16a-113f75252fc4","Type":"ContainerDied","Data":"9ee3de9d17a525dd1cd4e59abf08602d89d5a22ed3b1c2e58a02980b61fbfbc1"}
Oct 30 19:15:02.147672 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:02.147654    6014 scope.go:122] "RemoveContainer" containerID="9ee3de9d17a525dd1cd4e59abf08602d89d5a22ed3b1c2e58a02980b61fbfbc1"
Oct 30 19:15:02.149836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:02.149810    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz" event={"ID":"8f930fa4-482b-45e8-be8a-42bcbc4fa86f","Type":"ContainerStarted","Data":"90a510d07142c1d5f3ca31370b30bb8274d674e4e0dd30d3c59f96de7d046882"}
Oct 30 19:15:02.151617 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:02.151593    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg" event={"ID":"a9df751a-8363-49aa-ac14-e37ab07d6273","Type":"ContainerStarted","Data":"715683b6a2caf00546ac4cdc5f0234c36c94a2cda0597f655396df5ce9672e16"}
Oct 30 19:15:02.375277 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:02.375217    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6" podStartSLOduration=2.375202503 podStartE2EDuration="2.375202503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.259344071 +0000 UTC m=+2140.902511684" watchObservedRunningTime="2026-10-30 19:15:02.375202503 +0000 UTC m=+2141.018370130"
Oct 30 19:15:02.487323 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:02.487260    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg" podStartSLOduration=2.487242376 podStartE2EDuration="2.487242376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.485202655 +0000 UTC m=+2141.128370268" watchObservedRunningTime="2026-10-30 19:15:02.487242376 +0000 UTC m=+2141.130409989"
Oct 30 19:15:02.599661 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:02.599604    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz" podStartSLOduration=2.59958898 podStartE2EDuration="2.59958898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.597752839 +0000 UTC m=+2141.240920454" watchObservedRunningTime="2026-10-30 19:15:02.59958898 +0000 UTC m=+2141.242756605"
Oct 30 19:15:03.162558 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.162526    6014 generic.go:356] "Generic (PLEG): container finished" podID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" containerID="b631cb4e69f642e802809ff36ba17e3c9488b9f13188b78e13e940ef0a62dbf8" exitCode=60
Oct 30 19:15:03.162996 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.162620    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm" event={"ID":"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af","Type":"ContainerDied","Data":"b631cb4e69f642e802809ff36ba17e3c9488b9f13188b78e13e940ef0a62dbf8"}
Oct 30 19:15:03.162996 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.162709    6014 scope.go:122] "RemoveContainer" containerID="0c91e2d8cd575f4ff158a41450775c4d582cb11a11009f08d3d42657d54b88d3"
Oct 30 19:15:03.163122 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.163057    6014 scope.go:122] "RemoveContainer" containerID="b631cb4e69f642e802809ff36ba17e3c9488b9f13188b78e13e940ef0a62dbf8"
Oct 30 19:15:03.163517 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:03.163493    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-rpljm_odh-ai-gateway-infra(1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm" podUID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af"
Oct 30 19:15:03.164806 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.164781    6014 generic.go:356] "Generic (PLEG): container finished" podID="1a4f0028-19de-42ed-a16a-113f75252fc4" containerID="6288465e2faf651e72d8ae387e48cd750a8faac34719ca3cda9f2859856f0cba" exitCode=60
Oct 30 19:15:03.164885 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.164861    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h" event={"ID":"1a4f0028-19de-42ed-a16a-113f75252fc4","Type":"ContainerDied","Data":"6288465e2faf651e72d8ae387e48cd750a8faac34719ca3cda9f2859856f0cba"}
Oct 30 19:15:03.165204 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.165186    6014 scope.go:122] "RemoveContainer" containerID="6288465e2faf651e72d8ae387e48cd750a8faac34719ca3cda9f2859856f0cba"
Oct 30 19:15:03.165536 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:03.165517    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h_odh-ai-gateway-infra(1a4f0028-19de-42ed-a16a-113f75252fc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h" podUID="1a4f0028-19de-42ed-a16a-113f75252fc4"
Oct 30 19:15:03.166501 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.166482    6014 generic.go:356] "Generic (PLEG): container finished" podID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f" containerID="90a510d07142c1d5f3ca31370b30bb8274d674e4e0dd30d3c59f96de7d046882" exitCode=60
Oct 30 19:15:03.166646 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.166541    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz" event={"ID":"8f930fa4-482b-45e8-be8a-42bcbc4fa86f","Type":"ContainerDied","Data":"90a510d07142c1d5f3ca31370b30bb8274d674e4e0dd30d3c59f96de7d046882"}
Oct 30 19:15:03.166848 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.166827    6014 scope.go:122] "RemoveContainer" containerID="90a510d07142c1d5f3ca31370b30bb8274d674e4e0dd30d3c59f96de7d046882"
Oct 30 19:15:03.168444 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.168425    6014 generic.go:356] "Generic (PLEG): container finished" podID="a9df751a-8363-49aa-ac14-e37ab07d6273" containerID="715683b6a2caf00546ac4cdc5f0234c36c94a2cda0597f655396df5ce9672e16" exitCode=60
Oct 30 19:15:03.168539 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.168508    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg" event={"ID":"a9df751a-8363-49aa-ac14-e37ab07d6273","Type":"ContainerDied","Data":"715683b6a2caf00546ac4cdc5f0234c36c94a2cda0597f655396df5ce9672e16"}
Oct 30 19:15:03.168725 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.168705    6014 scope.go:122] "RemoveContainer" containerID="715683b6a2caf00546ac4cdc5f0234c36c94a2cda0597f655396df5ce9672e16"
Oct 30 19:15:03.170660 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.170636    6014 generic.go:356] "Generic (PLEG): container finished" podID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7" containerID="da9167484c52ea62e2918821a20c97217cdb296632a1e0c3f7a2b4b94c04ceb4" exitCode=60
Oct 30 19:15:03.170771 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.170686    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6" event={"ID":"b18e5c80-93bc-41a1-902a-d5f159e5c1c7","Type":"ContainerDied","Data":"da9167484c52ea62e2918821a20c97217cdb296632a1e0c3f7a2b4b94c04ceb4"}
Oct 30 19:15:03.170938 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.170925    6014 scope.go:122] "RemoveContainer" containerID="da9167484c52ea62e2918821a20c97217cdb296632a1e0c3f7a2b4b94c04ceb4"
Oct 30 19:15:03.295692 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:03.295674    6014 scope.go:122] "RemoveContainer" containerID="9ee3de9d17a525dd1cd4e59abf08602d89d5a22ed3b1c2e58a02980b61fbfbc1"
Oct 30 19:15:04.181069 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:04.181026    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6" event={"ID":"b18e5c80-93bc-41a1-902a-d5f159e5c1c7","Type":"ContainerStarted","Data":"8bfeb7e29f6d30d316b794fb1f8a8476ab8a8719fa895b06fc3511bdf0defc73"}
Oct 30 19:15:04.183234 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:04.183200    6014 scope.go:122] "RemoveContainer" containerID="b631cb4e69f642e802809ff36ba17e3c9488b9f13188b78e13e940ef0a62dbf8"
Oct 30 19:15:04.183599 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:04.183577    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-rpljm_odh-ai-gateway-infra(1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm" podUID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af"
Oct 30 19:15:04.184814 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:04.184793    6014 scope.go:122] "RemoveContainer" containerID="6288465e2faf651e72d8ae387e48cd750a8faac34719ca3cda9f2859856f0cba"
Oct 30 19:15:04.185168 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:04.185125    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h_odh-ai-gateway-infra(1a4f0028-19de-42ed-a16a-113f75252fc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h" podUID="1a4f0028-19de-42ed-a16a-113f75252fc4"
Oct 30 19:15:04.186428 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:04.186407    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz" event={"ID":"8f930fa4-482b-45e8-be8a-42bcbc4fa86f","Type":"ContainerStarted","Data":"2b2da45200dcf1fc205d54d693f6637dc7077238423e227dba1880af5c22dad1"}
Oct 30 19:15:04.188516 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:04.188494    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg" event={"ID":"a9df751a-8363-49aa-ac14-e37ab07d6273","Type":"ContainerStarted","Data":"efe0200c7201aa342103439ce9d2e4696b095cdfa664b21dc6c1dcc187460f65"}
Oct 30 19:15:05.198535 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.198506    6014 generic.go:356] "Generic (PLEG): container finished" podID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f" containerID="2b2da45200dcf1fc205d54d693f6637dc7077238423e227dba1880af5c22dad1" exitCode=60
Oct 30 19:15:05.199074 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.198591    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz" event={"ID":"8f930fa4-482b-45e8-be8a-42bcbc4fa86f","Type":"ContainerDied","Data":"2b2da45200dcf1fc205d54d693f6637dc7077238423e227dba1880af5c22dad1"}
Oct 30 19:15:05.199074 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.198662    6014 scope.go:122] "RemoveContainer" containerID="90a510d07142c1d5f3ca31370b30bb8274d674e4e0dd30d3c59f96de7d046882"
Oct 30 19:15:05.199074 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.198924    6014 scope.go:122] "RemoveContainer" containerID="2b2da45200dcf1fc205d54d693f6637dc7077238423e227dba1880af5c22dad1"
Oct 30 19:15:05.199400 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:05.199366    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz_odh-ai-gateway-infra(8f930fa4-482b-45e8-be8a-42bcbc4fa86f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz" podUID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f"
Oct 30 19:15:05.200587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.200569    6014 generic.go:356] "Generic (PLEG): container finished" podID="a9df751a-8363-49aa-ac14-e37ab07d6273" containerID="efe0200c7201aa342103439ce9d2e4696b095cdfa664b21dc6c1dcc187460f65" exitCode=60
Oct 30 19:15:05.200681 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.200649    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg" event={"ID":"a9df751a-8363-49aa-ac14-e37ab07d6273","Type":"ContainerDied","Data":"efe0200c7201aa342103439ce9d2e4696b095cdfa664b21dc6c1dcc187460f65"}
Oct 30 19:15:05.200934 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.200919    6014 scope.go:122] "RemoveContainer" containerID="efe0200c7201aa342103439ce9d2e4696b095cdfa664b21dc6c1dcc187460f65"
Oct 30 19:15:05.201329 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:05.201308    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg_odh-ai-gateway-infra(a9df751a-8363-49aa-ac14-e37ab07d6273)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg" podUID="a9df751a-8363-49aa-ac14-e37ab07d6273"
Oct 30 19:15:05.202470 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.202450    6014 generic.go:356] "Generic (PLEG): container finished" podID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7" containerID="8bfeb7e29f6d30d316b794fb1f8a8476ab8a8719fa895b06fc3511bdf0defc73" exitCode=60
Oct 30 19:15:05.202586 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.202551    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6" event={"ID":"b18e5c80-93bc-41a1-902a-d5f159e5c1c7","Type":"ContainerDied","Data":"8bfeb7e29f6d30d316b794fb1f8a8476ab8a8719fa895b06fc3511bdf0defc73"}
Oct 30 19:15:05.202785 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.202768    6014 scope.go:122] "RemoveContainer" containerID="8bfeb7e29f6d30d316b794fb1f8a8476ab8a8719fa895b06fc3511bdf0defc73"
Oct 30 19:15:05.203071 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:05.203051    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6_odh-ai-gateway-infra(b18e5c80-93bc-41a1-902a-d5f159e5c1c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6" podUID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7"
Oct 30 19:15:05.211549 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.211533    6014 scope.go:122] "RemoveContainer" containerID="715683b6a2caf00546ac4cdc5f0234c36c94a2cda0597f655396df5ce9672e16"
Oct 30 19:15:05.225484 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.225466    6014 scope.go:122] "RemoveContainer" containerID="da9167484c52ea62e2918821a20c97217cdb296632a1e0c3f7a2b4b94c04ceb4"
Oct 30 19:15:05.443252 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.443221    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"]
Oct 30 19:15:05.443513 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:05.443486    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerName="istio-proxy" containerID="cri-o://29ab93bcdc570cc990aae2e2b1c2740d7c72a9406e7bad811e9a5e66584ee2fe" gracePeriod=30
Oct 30 19:15:06.140059 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:06.140025    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:15:06.212445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:06.212394    6014 scope.go:122] "RemoveContainer" containerID="8bfeb7e29f6d30d316b794fb1f8a8476ab8a8719fa895b06fc3511bdf0defc73"
Oct 30 19:15:06.213009 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:06.212978    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6_odh-ai-gateway-infra(b18e5c80-93bc-41a1-902a-d5f159e5c1c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6" podUID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7"
Oct 30 19:15:06.221034 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:06.221008    6014 scope.go:122] "RemoveContainer" containerID="2b2da45200dcf1fc205d54d693f6637dc7077238423e227dba1880af5c22dad1"
Oct 30 19:15:06.221603 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:06.221568    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz_odh-ai-gateway-infra(8f930fa4-482b-45e8-be8a-42bcbc4fa86f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz" podUID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f"
Oct 30 19:15:06.222605 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:06.222587    6014 scope.go:122] "RemoveContainer" containerID="efe0200c7201aa342103439ce9d2e4696b095cdfa664b21dc6c1dcc187460f65"
Oct 30 19:15:06.222918 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:06.222900    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg_odh-ai-gateway-infra(a9df751a-8363-49aa-ac14-e37ab07d6273)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg" podUID="a9df751a-8363-49aa-ac14-e37ab07d6273"
Oct 30 19:15:10.817080 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.817050    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:15:10.883123 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.883082    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-data\") pod \"bd41d427-10c0-4e58-93fb-87d7d4296904\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") "
Oct 30 19:15:10.883348 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.883163    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-token\" (UniqueName: \"kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-token\") pod \"bd41d427-10c0-4e58-93fb-87d7d4296904\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") "
Oct 30 19:15:10.883348 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.883247    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-certs\") pod \"bd41d427-10c0-4e58-93fb-87d7d4296904\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") "
Oct 30 19:15:10.883461 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.883418    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-data" pod "bd41d427-10c0-4e58-93fb-87d7d4296904" (UID: "bd41d427-10c0-4e58-93fb-87d7d4296904"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:10.883616 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.883490    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-certs" pod "bd41d427-10c0-4e58-93fb-87d7d4296904" (UID: "bd41d427-10c0-4e58-93fb-87d7d4296904"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:10.883616 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.883502    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd41d427-10c0-4e58-93fb-87d7d4296904-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd41d427-10c0-4e58-93fb-87d7d4296904-istiod-ca-cert\") pod \"bd41d427-10c0-4e58-93fb-87d7d4296904\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") "
Oct 30 19:15:10.883616 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.883582    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-envoy\") pod \"bd41d427-10c0-4e58-93fb-87d7d4296904\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") "
Oct 30 19:15:10.883803 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.883641    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-podinfo\") pod \"bd41d427-10c0-4e58-93fb-87d7d4296904\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") "
Oct 30 19:15:10.883803 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.883714    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-socket\") pod \"bd41d427-10c0-4e58-93fb-87d7d4296904\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") "
Oct 30 19:15:10.883934 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.883797    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd41d427-10c0-4e58-93fb-87d7d4296904-istiod-ca-cert" pod "bd41d427-10c0-4e58-93fb-87d7d4296904" (UID: "bd41d427-10c0-4e58-93fb-87d7d4296904"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:10.883934 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.883804    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-credential-socket\") pod \"bd41d427-10c0-4e58-93fb-87d7d4296904\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") "
Oct 30 19:15:10.883934 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.883853    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-kube-api-access-v9fsq\" (UniqueName: \"kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-kube-api-access-v9fsq\") pod \"bd41d427-10c0-4e58-93fb-87d7d4296904\" (UID: \"bd41d427-10c0-4e58-93fb-87d7d4296904\") "
Oct 30 19:15:10.884080 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.883973    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-credential-socket" pod "bd41d427-10c0-4e58-93fb-87d7d4296904" (UID: "bd41d427-10c0-4e58-93fb-87d7d4296904"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:10.884127 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.884115    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-socket" pod "bd41d427-10c0-4e58-93fb-87d7d4296904" (UID: "bd41d427-10c0-4e58-93fb-87d7d4296904"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:10.885594 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.885565    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:10.885756 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.885598    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:10.885756 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.885615    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:10.885756 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.885627    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:10.885756 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.885641    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd41d427-10c0-4e58-93fb-87d7d4296904-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:10.886824 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.886801    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-kube-api-access-v9fsq" pod "bd41d427-10c0-4e58-93fb-87d7d4296904" (UID: "bd41d427-10c0-4e58-93fb-87d7d4296904"). InnerVolumeSpecName "kube-api-access-v9fsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:10.887042 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.887020    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-envoy" pod "bd41d427-10c0-4e58-93fb-87d7d4296904" (UID: "bd41d427-10c0-4e58-93fb-87d7d4296904"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:10.887104 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.887049    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-token" pod "bd41d427-10c0-4e58-93fb-87d7d4296904" (UID: "bd41d427-10c0-4e58-93fb-87d7d4296904"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:10.887250 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.887207    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-podinfo" pod "bd41d427-10c0-4e58-93fb-87d7d4296904" (UID: "bd41d427-10c0-4e58-93fb-87d7d4296904"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:10.988051 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.988018    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:10.988051 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.988049    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9fsq\" (UniqueName: \"kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-kube-api-access-v9fsq\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:10.988051 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.988062    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:10.988413 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:10.988071    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd41d427-10c0-4e58-93fb-87d7d4296904-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:11.288414 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:11.288381    6014 generic.go:356] "Generic (PLEG): container finished" podID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerID="29ab93bcdc570cc990aae2e2b1c2740d7c72a9406e7bad811e9a5e66584ee2fe" exitCode=0
Oct 30 19:15:11.288608 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:11.288453    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" event={"ID":"bd41d427-10c0-4e58-93fb-87d7d4296904","Type":"ContainerDied","Data":"29ab93bcdc570cc990aae2e2b1c2740d7c72a9406e7bad811e9a5e66584ee2fe"}
Oct 30 19:15:11.288608 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:11.288551    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db" event={"ID":"bd41d427-10c0-4e58-93fb-87d7d4296904","Type":"ContainerDied","Data":"ce76965677dab09b8b36d16a4257ff95448266a29c4d04c2db8d4c1921992569"}
Oct 30 19:15:11.288608 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:11.288582    6014 scope.go:122] "RemoveContainer" containerID="29ab93bcdc570cc990aae2e2b1c2740d7c72a9406e7bad811e9a5e66584ee2fe"
Oct 30 19:15:11.288831 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:11.288749    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"
Oct 30 19:15:11.304964 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:11.304941    6014 scope.go:122] "RemoveContainer" containerID="5b459a0bbde9234158a33ade757e6a107e957a6c1ae4ff7cbb9c94859548a498"
Oct 30 19:15:11.316402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:11.316372    6014 scope.go:122] "RemoveContainer" containerID="29ab93bcdc570cc990aae2e2b1c2740d7c72a9406e7bad811e9a5e66584ee2fe"
Oct 30 19:15:11.316713 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:11.316692    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ab93bcdc570cc990aae2e2b1c2740d7c72a9406e7bad811e9a5e66584ee2fe\": container with ID starting with 29ab93bcdc570cc990aae2e2b1c2740d7c72a9406e7bad811e9a5e66584ee2fe not found: ID does not exist" containerID="29ab93bcdc570cc990aae2e2b1c2740d7c72a9406e7bad811e9a5e66584ee2fe"
Oct 30 19:15:11.316788 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:11.316723    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ab93bcdc570cc990aae2e2b1c2740d7c72a9406e7bad811e9a5e66584ee2fe"} err="failed to get container status \"29ab93bcdc570cc990aae2e2b1c2740d7c72a9406e7bad811e9a5e66584ee2fe\": rpc error: code = NotFound desc = could not find container \"29ab93bcdc570cc990aae2e2b1c2740d7c72a9406e7bad811e9a5e66584ee2fe\": container with ID starting with 29ab93bcdc570cc990aae2e2b1c2740d7c72a9406e7bad811e9a5e66584ee2fe not found: ID does not exist"
Oct 30 19:15:11.316788 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:11.316743    6014 scope.go:122] "RemoveContainer" containerID="5b459a0bbde9234158a33ade757e6a107e957a6c1ae4ff7cbb9c94859548a498"
Oct 30 19:15:11.317031 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:11.317015    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b459a0bbde9234158a33ade757e6a107e957a6c1ae4ff7cbb9c94859548a498\": container with ID starting with 5b459a0bbde9234158a33ade757e6a107e957a6c1ae4ff7cbb9c94859548a498 not found: ID does not exist" containerID="5b459a0bbde9234158a33ade757e6a107e957a6c1ae4ff7cbb9c94859548a498"
Oct 30 19:15:11.317125 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:11.317106    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b459a0bbde9234158a33ade757e6a107e957a6c1ae4ff7cbb9c94859548a498"} err="failed to get container status \"5b459a0bbde9234158a33ade757e6a107e957a6c1ae4ff7cbb9c94859548a498\": rpc error: code = NotFound desc = could not find container \"5b459a0bbde9234158a33ade757e6a107e957a6c1ae4ff7cbb9c94859548a498\": container with ID starting with 5b459a0bbde9234158a33ade757e6a107e957a6c1ae4ff7cbb9c94859548a498 not found: ID does not exist"
Oct 30 19:15:11.463011 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:11.462979    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"]
Oct 30 19:15:11.507362 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:11.507329    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-322e97-openshift-default-856dffd7-kb7db"]
Oct 30 19:15:12.292576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:12.292539    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"]
Oct 30 19:15:12.341578 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:12.341535    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"]
Oct 30 19:15:13.478472 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:13.478439    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" path="/var/lib/kubelet/pods/bd41d427-10c0-4e58-93fb-87d7d4296904/volumes"
Oct 30 19:15:15.469195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:15.469119    6014 scope.go:122] "RemoveContainer" containerID="efe0200c7201aa342103439ce9d2e4696b095cdfa664b21dc6c1dcc187460f65"
Oct 30 19:15:15.469652 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:15.469313    6014 scope.go:122] "RemoveContainer" containerID="b631cb4e69f642e802809ff36ba17e3c9488b9f13188b78e13e940ef0a62dbf8"
Oct 30 19:15:16.327819 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:16.327778    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm" event={"ID":"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af","Type":"ContainerStarted","Data":"894144f629cc6c195a9cac59211bbdb6f5e2fcc188615c6d9a172374a539904a"}
Oct 30 19:15:16.330025 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:16.329996    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg" event={"ID":"a9df751a-8363-49aa-ac14-e37ab07d6273","Type":"ContainerStarted","Data":"e0d0fa4581aff24d1ff3d6080c608ae2c18f80ef75343d65cd670cbe9d8684b7"}
Oct 30 19:15:16.445777 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:16.445713    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm" podStartSLOduration=16.445695862 podStartE2EDuration="16.445695862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:16.440893492 +0000 UTC m=+2155.084061105" watchObservedRunningTime="2026-10-30 19:15:16.445695862 +0000 UTC m=+2155.088863479"
Oct 30 19:15:16.469608 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:16.469584    6014 scope.go:122] "RemoveContainer" containerID="6288465e2faf651e72d8ae387e48cd750a8faac34719ca3cda9f2859856f0cba"
Oct 30 19:15:17.340178 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:17.340116    6014 generic.go:356] "Generic (PLEG): container finished" podID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" containerID="894144f629cc6c195a9cac59211bbdb6f5e2fcc188615c6d9a172374a539904a" exitCode=60
Oct 30 19:15:17.340360 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:17.340229    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm" event={"ID":"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af","Type":"ContainerDied","Data":"894144f629cc6c195a9cac59211bbdb6f5e2fcc188615c6d9a172374a539904a"}
Oct 30 19:15:17.340360 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:17.340279    6014 scope.go:122] "RemoveContainer" containerID="b631cb4e69f642e802809ff36ba17e3c9488b9f13188b78e13e940ef0a62dbf8"
Oct 30 19:15:17.340728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:17.340713    6014 scope.go:122] "RemoveContainer" containerID="894144f629cc6c195a9cac59211bbdb6f5e2fcc188615c6d9a172374a539904a"
Oct 30 19:15:17.341087 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:17.341069    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-rpljm_odh-ai-gateway-infra(1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm" podUID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af"
Oct 30 19:15:17.343893 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:17.343854    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h" event={"ID":"1a4f0028-19de-42ed-a16a-113f75252fc4","Type":"ContainerStarted","Data":"a94bb6ff23e5a848cec26dba1b1c382ca03453abd95db77df84176c39f3b5538"}
Oct 30 19:15:17.345968 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:17.345949    6014 generic.go:356] "Generic (PLEG): container finished" podID="a9df751a-8363-49aa-ac14-e37ab07d6273" containerID="e0d0fa4581aff24d1ff3d6080c608ae2c18f80ef75343d65cd670cbe9d8684b7" exitCode=60
Oct 30 19:15:17.346090 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:17.346014    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg" event={"ID":"a9df751a-8363-49aa-ac14-e37ab07d6273","Type":"ContainerDied","Data":"e0d0fa4581aff24d1ff3d6080c608ae2c18f80ef75343d65cd670cbe9d8684b7"}
Oct 30 19:15:17.346428 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:17.346414    6014 scope.go:122] "RemoveContainer" containerID="e0d0fa4581aff24d1ff3d6080c608ae2c18f80ef75343d65cd670cbe9d8684b7"
Oct 30 19:15:17.346888 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:17.346731    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg_odh-ai-gateway-infra(a9df751a-8363-49aa-ac14-e37ab07d6273)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg" podUID="a9df751a-8363-49aa-ac14-e37ab07d6273"
Oct 30 19:15:17.356478 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:17.356459    6014 scope.go:122] "RemoveContainer" containerID="efe0200c7201aa342103439ce9d2e4696b095cdfa664b21dc6c1dcc187460f65"
Oct 30 19:15:17.452462 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:17.452432    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"]
Oct 30 19:15:17.566301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:17.566266    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"]
Oct 30 19:15:17.583580 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:17.583524    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h" podStartSLOduration=17.583504516 podStartE2EDuration="17.583504516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:17.580170026 +0000 UTC m=+2156.223337643" watchObservedRunningTime="2026-10-30 19:15:17.583504516 +0000 UTC m=+2156.226672130"
Oct 30 19:15:18.358865 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.358833    6014 generic.go:356] "Generic (PLEG): container finished" podID="1a4f0028-19de-42ed-a16a-113f75252fc4" containerID="a94bb6ff23e5a848cec26dba1b1c382ca03453abd95db77df84176c39f3b5538" exitCode=60
Oct 30 19:15:18.359016 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.358882    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h" event={"ID":"1a4f0028-19de-42ed-a16a-113f75252fc4","Type":"ContainerDied","Data":"a94bb6ff23e5a848cec26dba1b1c382ca03453abd95db77df84176c39f3b5538"}
Oct 30 19:15:18.359016 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.358950    6014 scope.go:122] "RemoveContainer" containerID="6288465e2faf651e72d8ae387e48cd750a8faac34719ca3cda9f2859856f0cba"
Oct 30 19:15:18.359442 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.359421    6014 scope.go:122] "RemoveContainer" containerID="a94bb6ff23e5a848cec26dba1b1c382ca03453abd95db77df84176c39f3b5538"
Oct 30 19:15:18.359854 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:18.359831    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h_odh-ai-gateway-infra(1a4f0028-19de-42ed-a16a-113f75252fc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h" podUID="1a4f0028-19de-42ed-a16a-113f75252fc4"
Oct 30 19:15:18.469608 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.469532    6014 scope.go:122] "RemoveContainer" containerID="8bfeb7e29f6d30d316b794fb1f8a8476ab8a8719fa895b06fc3511bdf0defc73"
Oct 30 19:15:18.472620 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.471008    6014 scope.go:122] "RemoveContainer" containerID="2b2da45200dcf1fc205d54d693f6637dc7077238423e227dba1880af5c22dad1"
Oct 30 19:15:18.600233 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.600200    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"]
Oct 30 19:15:18.618880 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.618803    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"
Oct 30 19:15:18.621444 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.621417    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"
Oct 30 19:15:18.815189 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.814815    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9df751a-8363-49aa-ac14-e37ab07d6273-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9df751a-8363-49aa-ac14-e37ab07d6273-maas-api-service-ca\") pod \"a9df751a-8363-49aa-ac14-e37ab07d6273\" (UID: \"a9df751a-8363-49aa-ac14-e37ab07d6273\") "
Oct 30 19:15:18.815189 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.814878    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-maas-api-service-ca\") pod \"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af\" (UID: \"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af\") "
Oct 30 19:15:18.815189 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.814926    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9df751a-8363-49aa-ac14-e37ab07d6273-kube-api-access-t8bdh\" (UniqueName: \"kubernetes.io/projected/a9df751a-8363-49aa-ac14-e37ab07d6273-kube-api-access-t8bdh\") pod \"a9df751a-8363-49aa-ac14-e37ab07d6273\" (UID: \"a9df751a-8363-49aa-ac14-e37ab07d6273\") "
Oct 30 19:15:18.815189 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.815048    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-kube-api-access-chcr5\" (UniqueName: \"kubernetes.io/projected/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-kube-api-access-chcr5\") pod \"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af\" (UID: \"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af\") "
Oct 30 19:15:18.816260 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.816230    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-maas-api-service-ca" pod "1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" (UID: "1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:18.816490 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.816463    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9df751a-8363-49aa-ac14-e37ab07d6273-maas-api-service-ca" pod "a9df751a-8363-49aa-ac14-e37ab07d6273" (UID: "a9df751a-8363-49aa-ac14-e37ab07d6273"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:18.818194 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.818134    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-kube-api-access-chcr5" pod "1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" (UID: "1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af"). InnerVolumeSpecName "kube-api-access-chcr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:18.818358 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.818341    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9df751a-8363-49aa-ac14-e37ab07d6273-kube-api-access-t8bdh" pod "a9df751a-8363-49aa-ac14-e37ab07d6273" (UID: "a9df751a-8363-49aa-ac14-e37ab07d6273"). InnerVolumeSpecName "kube-api-access-t8bdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:18.916811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.916709    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9df751a-8363-49aa-ac14-e37ab07d6273-maas-api-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:18.916811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.916745    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-maas-api-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:18.916811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.916760    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8bdh\" (UniqueName: \"kubernetes.io/projected/a9df751a-8363-49aa-ac14-e37ab07d6273-kube-api-access-t8bdh\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:18.916811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:18.916777    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chcr5\" (UniqueName: \"kubernetes.io/projected/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af-kube-api-access-chcr5\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.048701 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.048664    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"]
Oct 30 19:15:19.166689 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.166657    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"]
Oct 30 19:15:19.376023 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.375983    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm" event={"ID":"1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af","Type":"ContainerDied","Data":"41edcb09a9c286bc6f01cd76d278ff031e1cedf0a035cb10f7855dd219646479"}
Oct 30 19:15:19.376349 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.376328    6014 scope.go:122] "RemoveContainer" containerID="894144f629cc6c195a9cac59211bbdb6f5e2fcc188615c6d9a172374a539904a"
Oct 30 19:15:19.376616 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.376597    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"
Oct 30 19:15:19.397916 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.397881    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz" event={"ID":"8f930fa4-482b-45e8-be8a-42bcbc4fa86f","Type":"ContainerStarted","Data":"24a5cb99e2a5c9d56a362121761daf1387b187ac0ec21de015bbec00c603401a"}
Oct 30 19:15:19.403198 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.401272    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg" event={"ID":"a9df751a-8363-49aa-ac14-e37ab07d6273","Type":"ContainerDied","Data":"7911462025a08f8bcb511f7974be945056675625784d16c76a746a3dbf8dcd33"}
Oct 30 19:15:19.403198 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.401361    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"
Oct 30 19:15:19.405791 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.405755    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6" event={"ID":"b18e5c80-93bc-41a1-902a-d5f159e5c1c7","Type":"ContainerStarted","Data":"737d5da8e427a55d0987181c3c28cf339df890f8a02b1ffd50769729f206637a"}
Oct 30 19:15:19.412534 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.412512    6014 scope.go:122] "RemoveContainer" containerID="e0d0fa4581aff24d1ff3d6080c608ae2c18f80ef75343d65cd670cbe9d8684b7"
Oct 30 19:15:19.701131 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.701102    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"
Oct 30 19:15:19.729535 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.729504    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4f0028-19de-42ed-a16a-113f75252fc4-kube-api-access-pltz4\" (UniqueName: \"kubernetes.io/projected/1a4f0028-19de-42ed-a16a-113f75252fc4-kube-api-access-pltz4\") pod \"1a4f0028-19de-42ed-a16a-113f75252fc4\" (UID: \"1a4f0028-19de-42ed-a16a-113f75252fc4\") "
Oct 30 19:15:19.729717 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.729705    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a4f0028-19de-42ed-a16a-113f75252fc4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4f0028-19de-42ed-a16a-113f75252fc4-maas-api-service-ca\") pod \"1a4f0028-19de-42ed-a16a-113f75252fc4\" (UID: \"1a4f0028-19de-42ed-a16a-113f75252fc4\") "
Oct 30 19:15:19.730062 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.730037    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4f0028-19de-42ed-a16a-113f75252fc4-maas-api-service-ca" pod "1a4f0028-19de-42ed-a16a-113f75252fc4" (UID: "1a4f0028-19de-42ed-a16a-113f75252fc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:19.730279 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.730262    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4f0028-19de-42ed-a16a-113f75252fc4-maas-api-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.732259 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.732234    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4f0028-19de-42ed-a16a-113f75252fc4-kube-api-access-pltz4" pod "1a4f0028-19de-42ed-a16a-113f75252fc4" (UID: "1a4f0028-19de-42ed-a16a-113f75252fc4"). InnerVolumeSpecName "kube-api-access-pltz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:19.792081 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.792037    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"]
Oct 30 19:15:19.795501 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.795470    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rpljm"]
Oct 30 19:15:19.831934 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:19.831894    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pltz4\" (UniqueName: \"kubernetes.io/projected/1a4f0028-19de-42ed-a16a-113f75252fc4-kube-api-access-pltz4\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.015130 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.015041    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"]
Oct 30 19:15:20.018018 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.017995    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889795-8s2jg"]
Oct 30 19:15:20.415837 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.415803    6014 generic.go:356] "Generic (PLEG): container finished" podID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7" containerID="737d5da8e427a55d0987181c3c28cf339df890f8a02b1ffd50769729f206637a" exitCode=60
Oct 30 19:15:20.416013 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.415873    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6" event={"ID":"b18e5c80-93bc-41a1-902a-d5f159e5c1c7","Type":"ContainerDied","Data":"737d5da8e427a55d0987181c3c28cf339df890f8a02b1ffd50769729f206637a"}
Oct 30 19:15:20.416013 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.415939    6014 scope.go:122] "RemoveContainer" containerID="8bfeb7e29f6d30d316b794fb1f8a8476ab8a8719fa895b06fc3511bdf0defc73"
Oct 30 19:15:20.416406 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.416386    6014 scope.go:122] "RemoveContainer" containerID="737d5da8e427a55d0987181c3c28cf339df890f8a02b1ffd50769729f206637a"
Oct 30 19:15:20.416826 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:20.416802    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6_odh-ai-gateway-infra(b18e5c80-93bc-41a1-902a-d5f159e5c1c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6" podUID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7"
Oct 30 19:15:20.418736 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.418719    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"
Oct 30 19:15:20.418821 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.418747    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h" event={"ID":"1a4f0028-19de-42ed-a16a-113f75252fc4","Type":"ContainerDied","Data":"e19c40959293fafccb7f6fcc6f538b19ef5c2dcd9147c64cb4a4c369ab566b30"}
Oct 30 19:15:20.420857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.420839    6014 generic.go:356] "Generic (PLEG): container finished" podID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f" containerID="24a5cb99e2a5c9d56a362121761daf1387b187ac0ec21de015bbec00c603401a" exitCode=60
Oct 30 19:15:20.420959 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.420900    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz" event={"ID":"8f930fa4-482b-45e8-be8a-42bcbc4fa86f","Type":"ContainerDied","Data":"24a5cb99e2a5c9d56a362121761daf1387b187ac0ec21de015bbec00c603401a"}
Oct 30 19:15:20.421382 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.421366    6014 scope.go:122] "RemoveContainer" containerID="24a5cb99e2a5c9d56a362121761daf1387b187ac0ec21de015bbec00c603401a"
Oct 30 19:15:20.421692 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:20.421671    6014 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz_odh-ai-gateway-infra(8f930fa4-482b-45e8-be8a-42bcbc4fa86f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz" podUID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f"
Oct 30 19:15:20.429231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.429212    6014 scope.go:122] "RemoveContainer" containerID="a94bb6ff23e5a848cec26dba1b1c382ca03453abd95db77df84176c39f3b5538"
Oct 30 19:15:20.439591 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.439565    6014 scope.go:122] "RemoveContainer" containerID="2b2da45200dcf1fc205d54d693f6637dc7077238423e227dba1880af5c22dad1"
Oct 30 19:15:20.615163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.600193    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"]
Oct 30 19:15:20.646344 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.646313    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"]
Oct 30 19:15:20.649885 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.649857    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ae6a138d-29889795-jbm9h"]
Oct 30 19:15:20.690890 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:20.690819    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"]
Oct 30 19:15:21.486647 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.486603    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4f0028-19de-42ed-a16a-113f75252fc4" path="/var/lib/kubelet/pods/1a4f0028-19de-42ed-a16a-113f75252fc4/volumes"
Oct 30 19:15:21.497902 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.497867    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" path="/var/lib/kubelet/pods/1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af/volumes"
Oct 30 19:15:21.503176 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.498921    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9df751a-8363-49aa-ac14-e37ab07d6273" path="/var/lib/kubelet/pods/a9df751a-8363-49aa-ac14-e37ab07d6273/volumes"
Oct 30 19:15:21.639011 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.638982    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"
Oct 30 19:15:21.664976 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.664947    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-kube-api-access-7zfmx\" (UniqueName: \"kubernetes.io/projected/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-kube-api-access-7zfmx\") pod \"8f930fa4-482b-45e8-be8a-42bcbc4fa86f\" (UID: \"8f930fa4-482b-45e8-be8a-42bcbc4fa86f\") "
Oct 30 19:15:21.665163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.665096    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-maas-api-service-ca\") pod \"8f930fa4-482b-45e8-be8a-42bcbc4fa86f\" (UID: \"8f930fa4-482b-45e8-be8a-42bcbc4fa86f\") "
Oct 30 19:15:21.665418 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.665395    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-maas-api-service-ca" pod "8f930fa4-482b-45e8-be8a-42bcbc4fa86f" (UID: "8f930fa4-482b-45e8-be8a-42bcbc4fa86f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:21.665685 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.665670    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-maas-api-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.667455 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.667428    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-kube-api-access-7zfmx" pod "8f930fa4-482b-45e8-be8a-42bcbc4fa86f" (UID: "8f930fa4-482b-45e8-be8a-42bcbc4fa86f"). InnerVolumeSpecName "kube-api-access-7zfmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:21.670450 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.670426    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"
Oct 30 19:15:21.767199 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.767074    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-kube-api-access-vlpcr\" (UniqueName: \"kubernetes.io/projected/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-kube-api-access-vlpcr\") pod \"b18e5c80-93bc-41a1-902a-d5f159e5c1c7\" (UID: \"b18e5c80-93bc-41a1-902a-d5f159e5c1c7\") "
Oct 30 19:15:21.767396 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.767345    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-maas-api-service-ca\") pod \"b18e5c80-93bc-41a1-902a-d5f159e5c1c7\" (UID: \"b18e5c80-93bc-41a1-902a-d5f159e5c1c7\") "
Oct 30 19:15:21.767679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.767658    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-maas-api-service-ca" pod "b18e5c80-93bc-41a1-902a-d5f159e5c1c7" (UID: "b18e5c80-93bc-41a1-902a-d5f159e5c1c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:21.767842 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.767828    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-maas-api-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.767882 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.767845    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zfmx\" (UniqueName: \"kubernetes.io/projected/8f930fa4-482b-45e8-be8a-42bcbc4fa86f-kube-api-access-7zfmx\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.769448 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.769416    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-kube-api-access-vlpcr" pod "b18e5c80-93bc-41a1-902a-d5f159e5c1c7" (UID: "b18e5c80-93bc-41a1-902a-d5f159e5c1c7"). InnerVolumeSpecName "kube-api-access-vlpcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:21.869478 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:21.869444    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlpcr\" (UniqueName: \"kubernetes.io/projected/b18e5c80-93bc-41a1-902a-d5f159e5c1c7-kube-api-access-vlpcr\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.443471 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:22.443437    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"
Oct 30 19:15:22.443679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:22.443469    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6" event={"ID":"b18e5c80-93bc-41a1-902a-d5f159e5c1c7","Type":"ContainerDied","Data":"479c3b4adff50917700cf9c2fbe982528c946520dab1cc5aa003d58de738e21c"}
Oct 30 19:15:22.443679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:22.443533    6014 scope.go:122] "RemoveContainer" containerID="737d5da8e427a55d0987181c3c28cf339df890f8a02b1ffd50769729f206637a"
Oct 30 19:15:22.445113 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:22.445074    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz" event={"ID":"8f930fa4-482b-45e8-be8a-42bcbc4fa86f","Type":"ContainerDied","Data":"5409a28862409ee14593ce3f30fcab830e3f4889fe2bbed1df3905f12ce394cf"}
Oct 30 19:15:22.445263 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:22.445084    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"
Oct 30 19:15:22.455288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:22.455183    6014 scope.go:122] "RemoveContainer" containerID="24a5cb99e2a5c9d56a362121761daf1387b187ac0ec21de015bbec00c603401a"
Oct 30 19:15:22.612590 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:22.612550    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"]
Oct 30 19:15:22.661371 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:22.661340    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-24e88a-29889795-sl8c6"]
Oct 30 19:15:22.728784 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:22.728690    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"]
Oct 30 19:15:22.733290 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:22.733264    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889795-gxkgz"]
Oct 30 19:15:23.269282 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:23.269246    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:15:23.478487 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:23.478450    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f" path="/var/lib/kubelet/pods/8f930fa4-482b-45e8-be8a-42bcbc4fa86f/volumes"
Oct 30 19:15:23.479260 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:23.479240    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7" path="/var/lib/kubelet/pods/b18e5c80-93bc-41a1-902a-d5f159e5c1c7/volumes"
Oct 30 19:15:24.049396 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:24.049352    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk" podUID="fe3f0683-26aa-408e-98d2-97d34bba0d14" containerName="payload-processing" containerID="cri-o://478079fb93390826deb769516568bddaef7996d9733160bb57e715b62d18d80f" gracePeriod=25
Oct 30 19:15:24.167473 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:24.167426    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg" podUID="7ee7e187-7e2a-4105-a4d0-1fe11a63464f" containerName="payload-pre-processing" containerID="cri-o://24414dbda1894e4effc481934606d8b1c710c724f3612a40bc2bf2011a353c88" gracePeriod=25
Oct 30 19:15:27.316809 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.316768    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" containerID="cri-o://7a178c73a4f4ce7945453f7fb76a903643fc6a8a06cb04d428072aa2c0e1e175" gracePeriod=45
Oct 30 19:15:27.363870 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.363824    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main" containerID="cri-o://d5ff7ef6c10785f135b116c16e9d26a8e26b41a576d9232a16aaadc5f03f9ab1" gracePeriod=45
Oct 30 19:15:27.438679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.438632    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": dial tcp 10.134.0.80:8000: connect: connection refused"
Oct 30 19:15:27.500858 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.500818    6014 generic.go:356] "Generic (PLEG): container finished" podID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerID="7a178c73a4f4ce7945453f7fb76a903643fc6a8a06cb04d428072aa2c0e1e175" exitCode=0
Oct 30 19:15:27.501064 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.501019    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" event={"ID":"1fb951bd-2257-46f6-8adc-50f8fa5f7082","Type":"ContainerDied","Data":"7a178c73a4f4ce7945453f7fb76a903643fc6a8a06cb04d428072aa2c0e1e175"}
Oct 30 19:15:27.503442 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.503331    6014 generic.go:356] "Generic (PLEG): container finished" podID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerID="d5ff7ef6c10785f135b116c16e9d26a8e26b41a576d9232a16aaadc5f03f9ab1" exitCode=0
Oct 30 19:15:27.503442 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.503376    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" event={"ID":"41ac9daa-792e-4206-baa8-db8f3c4ccce9","Type":"ContainerDied","Data":"d5ff7ef6c10785f135b116c16e9d26a8e26b41a576d9232a16aaadc5f03f9ab1"}
Oct 30 19:15:27.668391 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.668360    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:15:27.710931 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.710892    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 19:15:27.759027 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.758989    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-dshm\") pod \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") "
Oct 30 19:15:27.759237 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.759068    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tmp-dir\") pod \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") "
Oct 30 19:15:27.759237 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.759202    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb951bd-2257-46f6-8adc-50f8fa5f7082-kube-api-access-ghnxn\" (UniqueName: \"kubernetes.io/projected/1fb951bd-2257-46f6-8adc-50f8fa5f7082-kube-api-access-ghnxn\") pod \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") "
Oct 30 19:15:27.759342 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.759290    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-model-cache\") pod \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") "
Oct 30 19:15:27.759342 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.759313    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tmp-dir" pod "1fb951bd-2257-46f6-8adc-50f8fa5f7082" (UID: "1fb951bd-2257-46f6-8adc-50f8fa5f7082"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.759451 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.759347    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tls-certs\") pod \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") "
Oct 30 19:15:27.759503 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.759480    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-home\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-home\") pod \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\" (UID: \"1fb951bd-2257-46f6-8adc-50f8fa5f7082\") "
Oct 30 19:15:27.759557 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.759514    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-model-cache" pod "1fb951bd-2257-46f6-8adc-50f8fa5f7082" (UID: "1fb951bd-2257-46f6-8adc-50f8fa5f7082"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.759863 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.759815    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-home" pod "1fb951bd-2257-46f6-8adc-50f8fa5f7082" (UID: "1fb951bd-2257-46f6-8adc-50f8fa5f7082"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.760232 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.760214    6014 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-model-cache\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.760232 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.760233    6014 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-home\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.760369 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.760244    6014 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tmp-dir\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.761637 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.761612    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-dshm" pod "1fb951bd-2257-46f6-8adc-50f8fa5f7082" (UID: "1fb951bd-2257-46f6-8adc-50f8fa5f7082"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.761729 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.761710    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb951bd-2257-46f6-8adc-50f8fa5f7082-kube-api-access-ghnxn" pod "1fb951bd-2257-46f6-8adc-50f8fa5f7082" (UID: "1fb951bd-2257-46f6-8adc-50f8fa5f7082"). InnerVolumeSpecName "kube-api-access-ghnxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:27.761950 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.761929    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tls-certs" pod "1fb951bd-2257-46f6-8adc-50f8fa5f7082" (UID: "1fb951bd-2257-46f6-8adc-50f8fa5f7082"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:27.861183 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.861077    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tls-certs\" (UniqueName: \"kubernetes.io/secret/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tls-certs\") pod \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") "
Oct 30 19:15:27.861183 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.861120    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-model-cache\") pod \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") "
Oct 30 19:15:27.861424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.861203    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tmp-dir\") pod \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") "
Oct 30 19:15:27.861424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.861224    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-dshm\") pod \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") "
Oct 30 19:15:27.861424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.861245    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ac9daa-792e-4206-baa8-db8f3c4ccce9-kube-api-access-wd7p7\" (UniqueName: \"kubernetes.io/projected/41ac9daa-792e-4206-baa8-db8f3c4ccce9-kube-api-access-wd7p7\") pod \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") "
Oct 30 19:15:27.861424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.861324    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-home\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-home\") pod \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\" (UID: \"41ac9daa-792e-4206-baa8-db8f3c4ccce9\") "
Oct 30 19:15:27.861632 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.861495    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tmp-dir" pod "41ac9daa-792e-4206-baa8-db8f3c4ccce9" (UID: "41ac9daa-792e-4206-baa8-db8f3c4ccce9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.861632 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.861574    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-model-cache" pod "41ac9daa-792e-4206-baa8-db8f3c4ccce9" (UID: "41ac9daa-792e-4206-baa8-db8f3c4ccce9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.861767 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.861738    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-home" pod "41ac9daa-792e-4206-baa8-db8f3c4ccce9" (UID: "41ac9daa-792e-4206-baa8-db8f3c4ccce9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.862259 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.862231    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghnxn\" (UniqueName: \"kubernetes.io/projected/1fb951bd-2257-46f6-8adc-50f8fa5f7082-kube-api-access-ghnxn\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.862259 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.862256    6014 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-model-cache\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.862434 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.862271    6014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb951bd-2257-46f6-8adc-50f8fa5f7082-tls-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.862434 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.862285    6014 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tmp-dir\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.862434 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.862294    6014 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-home\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.862434 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.862302    6014 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb951bd-2257-46f6-8adc-50f8fa5f7082-dshm\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.863696 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.863667    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tls-certs" pod "41ac9daa-792e-4206-baa8-db8f3c4ccce9" (UID: "41ac9daa-792e-4206-baa8-db8f3c4ccce9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:27.863900 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.863881    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-dshm" pod "41ac9daa-792e-4206-baa8-db8f3c4ccce9" (UID: "41ac9daa-792e-4206-baa8-db8f3c4ccce9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.864035 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.864018    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ac9daa-792e-4206-baa8-db8f3c4ccce9-kube-api-access-wd7p7" pod "41ac9daa-792e-4206-baa8-db8f3c4ccce9" (UID: "41ac9daa-792e-4206-baa8-db8f3c4ccce9"). InnerVolumeSpecName "kube-api-access-wd7p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:27.964290 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.964249    6014 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ac9daa-792e-4206-baa8-db8f3c4ccce9-dshm\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.964290 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.964292    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd7p7\" (UniqueName: \"kubernetes.io/projected/41ac9daa-792e-4206-baa8-db8f3c4ccce9-kube-api-access-wd7p7\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.964496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:27.964309    6014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ac9daa-792e-4206-baa8-db8f3c4ccce9-tls-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:28.511811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:28.511782    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"
Oct 30 19:15:28.511811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:28.511805    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs" event={"ID":"41ac9daa-792e-4206-baa8-db8f3c4ccce9","Type":"ContainerDied","Data":"983b6c4bfeea13d821bcee5de01f13d8a99f62e51c77e61f9120a4e4c864bd5c"}
Oct 30 19:15:28.512377 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:28.511853    6014 scope.go:122] "RemoveContainer" containerID="d5ff7ef6c10785f135b116c16e9d26a8e26b41a576d9232a16aaadc5f03f9ab1"
Oct 30 19:15:28.513592 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:28.513560    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx" event={"ID":"1fb951bd-2257-46f6-8adc-50f8fa5f7082","Type":"ContainerDied","Data":"4840062f3a61cf7b39280f7cb8e8eea4566d570eaffa1d3c7c72810ef5b52449"}
Oct 30 19:15:28.513592 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:28.513578    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"
Oct 30 19:15:28.523089 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:28.523067    6014 scope.go:122] "RemoveContainer" containerID="7a178c73a4f4ce7945453f7fb76a903643fc6a8a06cb04d428072aa2c0e1e175"
Oct 30 19:15:28.678317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:28.678191    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"]
Oct 30 19:15:28.725795 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:28.725752    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/distinct-2-w0-f5763a-kserve-56887d4dc8-pwjqx"]
Oct 30 19:15:28.836725 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:28.836691    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"]
Oct 30 19:15:28.843275 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:28.843244    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/premium-w0-f5763a-kserve-ccb895d7d-6xnqs"]
Oct 30 19:15:29.477722 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:29.477688    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" path="/var/lib/kubelet/pods/1fb951bd-2257-46f6-8adc-50f8fa5f7082/volumes"
Oct 30 19:15:29.478185 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:29.478167    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" path="/var/lib/kubelet/pods/41ac9daa-792e-4206-baa8-db8f3c4ccce9/volumes"
Oct 30 19:15:30.311554 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:30.311519    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk" podUID="fe3f0683-26aa-408e-98d2-97d34bba0d14" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:31.319567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:31.319521    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg" podUID="7ee7e187-7e2a-4105-a4d0-1fe11a63464f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:34.233644 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.233599    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"]
Oct 30 19:15:34.234587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234563    6014 cpu_manager.go:414] "containerMap: removing container" podUID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7" containerName="cleanup"
Oct 30 19:15:34.234587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234587    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7" containerName="cleanup"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234607    6014 cpu_manager.go:414] "containerMap: removing container" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234613    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234624    6014 cpu_manager.go:414] "containerMap: removing container" podUID="a9df751a-8363-49aa-ac14-e37ab07d6273" containerName="cleanup"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234630    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9df751a-8363-49aa-ac14-e37ab07d6273" containerName="cleanup"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234642    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" containerName="cleanup"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234649    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" containerName="cleanup"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234665    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1a4f0028-19de-42ed-a16a-113f75252fc4" containerName="cleanup"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234672    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4f0028-19de-42ed-a16a-113f75252fc4" containerName="cleanup"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234686    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" containerName="cleanup"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234693    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" containerName="cleanup"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234705    6014 cpu_manager.go:414] "containerMap: removing container" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerName="istio-proxy"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234712    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerName="istio-proxy"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234729    6014 cpu_manager.go:414] "containerMap: removing container" podUID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7" containerName="cleanup"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234737    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7" containerName="cleanup"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234750    6014 cpu_manager.go:414] "containerMap: removing container" podUID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f" containerName="cleanup"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234757    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f" containerName="cleanup"
Oct 30 19:15:34.234773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234775    6014 cpu_manager.go:414] "containerMap: removing container" podUID="a9df751a-8363-49aa-ac14-e37ab07d6273" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234782    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9df751a-8363-49aa-ac14-e37ab07d6273" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234794    6014 cpu_manager.go:414] "containerMap: removing container" podUID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234801    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234814    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234822    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234836    6014 cpu_manager.go:414] "containerMap: removing container" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerName="istio-proxy"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234842    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerName="istio-proxy"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234853    6014 cpu_manager.go:414] "containerMap: removing container" podUID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234860    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234875    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1a4f0028-19de-42ed-a16a-113f75252fc4" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234882    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4f0028-19de-42ed-a16a-113f75252fc4" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234901    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.234908    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235111    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9df751a-8363-49aa-ac14-e37ab07d6273" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235127    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4f0028-19de-42ed-a16a-113f75252fc4" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235178    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235191    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ac9daa-792e-4206-baa8-db8f3c4ccce9" containerName="main"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235204    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9df751a-8363-49aa-ac14-e37ab07d6273" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235211    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235221    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4f0028-19de-42ed-a16a-113f75252fc4" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235229    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235238    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerName="istio-proxy"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235249    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235259    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9df751a-8363-49aa-ac14-e37ab07d6273" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235270    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235281    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb951bd-2257-46f6-8adc-50f8fa5f7082" containerName="main"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235290    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd41d427-10c0-4e58-93fb-87d7d4296904" containerName="istio-proxy"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235298    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f930fa4-482b-45e8-be8a-42bcbc4fa86f" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235307    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4f0028-19de-42ed-a16a-113f75252fc4" containerName="cleanup"
Oct 30 19:15:34.235679 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.235317    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7" containerName="cleanup"
Oct 30 19:15:34.241117 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.241094    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.283185 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.283155    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"]
Oct 30 19:15:34.283891 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.283876    6014 cpu_manager.go:414] "containerMap: removing container" podUID="a9df751a-8363-49aa-ac14-e37ab07d6273" containerName="cleanup"
Oct 30 19:15:34.283937 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.283893    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9df751a-8363-49aa-ac14-e37ab07d6273" containerName="cleanup"
Oct 30 19:15:34.283937 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.283916    6014 cpu_manager.go:414] "containerMap: removing container" podUID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7" containerName="cleanup"
Oct 30 19:15:34.283937 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.283921    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7" containerName="cleanup"
Oct 30 19:15:34.284040 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.283939    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1a4f0028-19de-42ed-a16a-113f75252fc4" containerName="cleanup"
Oct 30 19:15:34.284040 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.283944    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4f0028-19de-42ed-a16a-113f75252fc4" containerName="cleanup"
Oct 30 19:15:34.284117 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.284040    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf54a96-1b68-4a93-b3d7-f6cbdc7d76af" containerName="cleanup"
Oct 30 19:15:34.284117 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.284053    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18e5c80-93bc-41a1-902a-d5f159e5c1c7" containerName="cleanup"
Oct 30 19:15:34.288219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.288196    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.341958 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.341926    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"]
Oct 30 19:15:34.361546 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.361496    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmb6w\" (UniqueName: \"kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-kube-api-access-pmb6w\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.361727 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.361561    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.361829 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.361714    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da67831f-c12e-47db-ae2b-6a9276287b74-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.361829 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.361788    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.361952 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.361921    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.362045 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.362026    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.362103 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.362091    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.362190 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.362116    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.362355 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.362332    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da67831f-c12e-47db-ae2b-6a9276287b74-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.393265 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.393229    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"]
Oct 30 19:15:34.463746 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.463699    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.463902 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.463777    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.464012 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.463930    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.464251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464229    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.464328 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464311    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.464377 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464341    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.464416 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464398    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.464475 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464459    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.464523 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464510    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da67831f-c12e-47db-ae2b-6a9276287b74-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.464559 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464550    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmb6w\" (UniqueName: \"kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-kube-api-access-pmb6w\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.464602 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464591    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.464661 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464645    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ssnk\" (UniqueName: \"kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-kube-api-access-2ssnk\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.464728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464719    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da67831f-c12e-47db-ae2b-6a9276287b74-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.464796 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464780    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.464853 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464814    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.464901 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464874    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.464966 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464931    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.464966 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464940    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.464966 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.464960    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.465310 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.465058    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.465499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.465473    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.466008 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.465981    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.466462 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.466432    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da67831f-c12e-47db-ae2b-6a9276287b74-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.467555 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.467525    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.468094 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.468073    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da67831f-c12e-47db-ae2b-6a9276287b74-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.523595 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.523526    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.523742 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.523635    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmb6w\" (UniqueName: \"kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-kube-api-access-pmb6w\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.558346 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.558308    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:34.565789 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.565762    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.565938 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.565809    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.565938 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.565842    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.565938 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.565885    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.566105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.565966    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ssnk\" (UniqueName: \"kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-kube-api-access-2ssnk\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.566105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.566071    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.566241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.566171    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.566241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.566191    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.566241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.566223    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.566395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.566266    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.566395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.566313    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.566512 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.566442    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.566572 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.566536    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.566799 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.566780    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.569120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.569087    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.569268 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.569189    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.628504 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.628428    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ssnk\" (UniqueName: \"kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-kube-api-access-2ssnk\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.629054 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.629027    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:34.902719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:34.902687    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:35.014071 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.014036    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"]
Oct 30 19:15:35.016040 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.016000    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:15:35.016159 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.016075    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:15:35.016159 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.016105    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:15:35.285797 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.285765    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"]
Oct 30 19:15:35.288405 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:15:35.288380    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84ead0c_c769_4f58_bc0f_e5d7528ae907.slice/crio-82c73574dcda053a9036744f7a02b8efbfcba90a7ae6badb89d521700d161888 WatchSource:0}: Error finding container 82c73574dcda053a9036744f7a02b8efbfcba90a7ae6badb89d521700d161888: Status 404 returned error can't find the container with id 82c73574dcda053a9036744f7a02b8efbfcba90a7ae6badb89d521700d161888
Oct 30 19:15:35.291222 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.291187    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:15:35.291322 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.291264    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:15:35.291322 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.291296    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:15:35.592165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.579537    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm" event={"ID":"da67831f-c12e-47db-ae2b-6a9276287b74","Type":"ContainerStarted","Data":"739b0abed625d038576d8792f4e8152f235b9dfa0a015f92e7fba509bc3a5346"}
Oct 30 19:15:35.592165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.579610    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm" event={"ID":"da67831f-c12e-47db-ae2b-6a9276287b74","Type":"ContainerStarted","Data":"85ace61612ab1cc555e512bec37b09508b43c6aefcae309a0e8c9a4c5b44aaa3"}
Oct 30 19:15:35.595902 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.594809    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t" event={"ID":"f84ead0c-c769-4f58-bc0f-e5d7528ae907","Type":"ContainerStarted","Data":"345454db6c78678434492d6cbdd97ce4008e8bfb856c85b5118776a3662d2f53"}
Oct 30 19:15:35.595902 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.594879    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t" event={"ID":"f84ead0c-c769-4f58-bc0f-e5d7528ae907","Type":"ContainerStarted","Data":"82c73574dcda053a9036744f7a02b8efbfcba90a7ae6badb89d521700d161888"}
Oct 30 19:15:35.699071 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.699011    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm" podStartSLOduration=1.698996621 podStartE2EDuration="1.698996621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:35.696248339 +0000 UTC m=+2174.339415955" watchObservedRunningTime="2026-10-30 19:15:35.698996621 +0000 UTC m=+2174.342164234"
Oct 30 19:15:35.821523 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.821472    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t" podStartSLOduration=1.821451905 podStartE2EDuration="1.821451905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:35.817509443 +0000 UTC m=+2174.460677070" watchObservedRunningTime="2026-10-30 19:15:35.821451905 +0000 UTC m=+2174.464619521"
Oct 30 19:15:35.905224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:35.903291    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:36.559595 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:36.559560    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:36.573177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:36.572286    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 19:15:36.573177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:36.572352    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm" podUID="da67831f-c12e-47db-ae2b-6a9276287b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 19:15:36.905590 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:36.905544    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:15:36.905760 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:36.905622    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t" podUID="f84ead0c-c769-4f58-bc0f-e5d7528ae907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 19:15:37.563635 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:37.563595    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 19:15:37.564238 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:37.564205    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm" podUID="da67831f-c12e-47db-ae2b-6a9276287b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 19:15:37.903230 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:37.903092    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:15:37.903230 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:37.903179    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t" podUID="f84ead0c-c769-4f58-bc0f-e5d7528ae907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 19:15:38.581168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:38.578297    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 19:15:38.581168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:38.578374    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm" podUID="da67831f-c12e-47db-ae2b-6a9276287b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 19:15:38.905102 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:38.905015    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:15:38.905388 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:38.905356    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t" podUID="f84ead0c-c769-4f58-bc0f-e5d7528ae907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 19:15:39.567165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:39.567018    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 19:15:39.567165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:39.567088    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm" podUID="da67831f-c12e-47db-ae2b-6a9276287b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 19:15:39.904153 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:39.904062    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:15:39.904504 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:39.904131    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t" podUID="f84ead0c-c769-4f58-bc0f-e5d7528ae907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 19:15:40.310887 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:40.310798    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk" podUID="fe3f0683-26aa-408e-98d2-97d34bba0d14" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:40.576164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:40.574299    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 19:15:40.576164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:40.574366    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm" podUID="da67831f-c12e-47db-ae2b-6a9276287b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 19:15:40.904110 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:40.903684    6014 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:15:40.904110 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:40.903734    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t" podUID="f84ead0c-c769-4f58-bc0f-e5d7528ae907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 19:15:41.320233 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:41.320199    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg" podUID="7ee7e187-7e2a-4105-a4d0-1fe11a63464f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:41.643090 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:41.642994    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:41.644363 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:41.644340    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:41.645377 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:41.645306    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:15:41.909634 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:41.909548    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:41.910121 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:41.909927    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:41.911084 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:41.911063    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:15:44.336979 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.336949    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"
Oct 30 19:15:44.360663 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.360261    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe3f0683-26aa-408e-98d2-97d34bba0d14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe3f0683-26aa-408e-98d2-97d34bba0d14-plugins-config-volume\") pod \"fe3f0683-26aa-408e-98d2-97d34bba0d14\" (UID: \"fe3f0683-26aa-408e-98d2-97d34bba0d14\") "
Oct 30 19:15:44.360663 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.360442    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe3f0683-26aa-408e-98d2-97d34bba0d14-kube-api-access-mrwxg\" (UniqueName: \"kubernetes.io/projected/fe3f0683-26aa-408e-98d2-97d34bba0d14-kube-api-access-mrwxg\") pod \"fe3f0683-26aa-408e-98d2-97d34bba0d14\" (UID: \"fe3f0683-26aa-408e-98d2-97d34bba0d14\") "
Oct 30 19:15:44.361603 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.361279    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3f0683-26aa-408e-98d2-97d34bba0d14-plugins-config-volume" pod "fe3f0683-26aa-408e-98d2-97d34bba0d14" (UID: "fe3f0683-26aa-408e-98d2-97d34bba0d14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:44.362242 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.362210    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe3f0683-26aa-408e-98d2-97d34bba0d14-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.369748 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.369721    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3f0683-26aa-408e-98d2-97d34bba0d14-kube-api-access-mrwxg" pod "fe3f0683-26aa-408e-98d2-97d34bba0d14" (UID: "fe3f0683-26aa-408e-98d2-97d34bba0d14"). InnerVolumeSpecName "kube-api-access-mrwxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:44.443462 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.443431    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"
Oct 30 19:15:44.463532 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.463497    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-kube-api-access-c79g8\" (UniqueName: \"kubernetes.io/projected/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-kube-api-access-c79g8\") pod \"7ee7e187-7e2a-4105-a4d0-1fe11a63464f\" (UID: \"7ee7e187-7e2a-4105-a4d0-1fe11a63464f\") "
Oct 30 19:15:44.463746 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.463730    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-plugins-config-volume\") pod \"7ee7e187-7e2a-4105-a4d0-1fe11a63464f\" (UID: \"7ee7e187-7e2a-4105-a4d0-1fe11a63464f\") "
Oct 30 19:15:44.464158 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.464113    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-plugins-config-volume" pod "7ee7e187-7e2a-4105-a4d0-1fe11a63464f" (UID: "7ee7e187-7e2a-4105-a4d0-1fe11a63464f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:44.464368 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.464351    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.464450 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.464369    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrwxg\" (UniqueName: \"kubernetes.io/projected/fe3f0683-26aa-408e-98d2-97d34bba0d14-kube-api-access-mrwxg\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.466382 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.466341    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-kube-api-access-c79g8" pod "7ee7e187-7e2a-4105-a4d0-1fe11a63464f" (UID: "7ee7e187-7e2a-4105-a4d0-1fe11a63464f"). InnerVolumeSpecName "kube-api-access-c79g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:44.566456 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.566424    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c79g8\" (UniqueName: \"kubernetes.io/projected/7ee7e187-7e2a-4105-a4d0-1fe11a63464f-kube-api-access-c79g8\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.676051 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.676014    6014 generic.go:356] "Generic (PLEG): container finished" podID="fe3f0683-26aa-408e-98d2-97d34bba0d14" containerID="478079fb93390826deb769516568bddaef7996d9733160bb57e715b62d18d80f" exitCode=0
Oct 30 19:15:44.676273 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.676065    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"
Oct 30 19:15:44.676273 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.676107    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk" event={"ID":"fe3f0683-26aa-408e-98d2-97d34bba0d14","Type":"ContainerDied","Data":"478079fb93390826deb769516568bddaef7996d9733160bb57e715b62d18d80f"}
Oct 30 19:15:44.676273 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.676194    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk" event={"ID":"fe3f0683-26aa-408e-98d2-97d34bba0d14","Type":"ContainerDied","Data":"6027679de4c93140058bc6d85a9113fde4c8bc6f5b872a518f2d08ca27325d6e"}
Oct 30 19:15:44.676273 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.676223    6014 scope.go:122] "RemoveContainer" containerID="478079fb93390826deb769516568bddaef7996d9733160bb57e715b62d18d80f"
Oct 30 19:15:44.678359 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.678338    6014 generic.go:356] "Generic (PLEG): container finished" podID="7ee7e187-7e2a-4105-a4d0-1fe11a63464f" containerID="24414dbda1894e4effc481934606d8b1c710c724f3612a40bc2bf2011a353c88" exitCode=0
Oct 30 19:15:44.678434 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.678420    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg" event={"ID":"7ee7e187-7e2a-4105-a4d0-1fe11a63464f","Type":"ContainerDied","Data":"24414dbda1894e4effc481934606d8b1c710c724f3612a40bc2bf2011a353c88"}
Oct 30 19:15:44.678469 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.678462    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg" event={"ID":"7ee7e187-7e2a-4105-a4d0-1fe11a63464f","Type":"ContainerDied","Data":"b3b3d229e1989c5f1be0edb36439f5e6536820f64dfe946bedfaef59da7fb726"}
Oct 30 19:15:44.678536 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.678523    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"
Oct 30 19:15:44.688630 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.688608    6014 scope.go:122] "RemoveContainer" containerID="478079fb93390826deb769516568bddaef7996d9733160bb57e715b62d18d80f"
Oct 30 19:15:44.688973 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:44.688950    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"478079fb93390826deb769516568bddaef7996d9733160bb57e715b62d18d80f\": container with ID starting with 478079fb93390826deb769516568bddaef7996d9733160bb57e715b62d18d80f not found: ID does not exist" containerID="478079fb93390826deb769516568bddaef7996d9733160bb57e715b62d18d80f"
Oct 30 19:15:44.689062 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.688984    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478079fb93390826deb769516568bddaef7996d9733160bb57e715b62d18d80f"} err="failed to get container status \"478079fb93390826deb769516568bddaef7996d9733160bb57e715b62d18d80f\": rpc error: code = NotFound desc = could not find container \"478079fb93390826deb769516568bddaef7996d9733160bb57e715b62d18d80f\": container with ID starting with 478079fb93390826deb769516568bddaef7996d9733160bb57e715b62d18d80f not found: ID does not exist"
Oct 30 19:15:44.689062 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.689007    6014 scope.go:122] "RemoveContainer" containerID="24414dbda1894e4effc481934606d8b1c710c724f3612a40bc2bf2011a353c88"
Oct 30 19:15:44.698387 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.698366    6014 scope.go:122] "RemoveContainer" containerID="24414dbda1894e4effc481934606d8b1c710c724f3612a40bc2bf2011a353c88"
Oct 30 19:15:44.698679 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:44.698660    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24414dbda1894e4effc481934606d8b1c710c724f3612a40bc2bf2011a353c88\": container with ID starting with 24414dbda1894e4effc481934606d8b1c710c724f3612a40bc2bf2011a353c88 not found: ID does not exist" containerID="24414dbda1894e4effc481934606d8b1c710c724f3612a40bc2bf2011a353c88"
Oct 30 19:15:44.698734 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.698688    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24414dbda1894e4effc481934606d8b1c710c724f3612a40bc2bf2011a353c88"} err="failed to get container status \"24414dbda1894e4effc481934606d8b1c710c724f3612a40bc2bf2011a353c88\": rpc error: code = NotFound desc = could not find container \"24414dbda1894e4effc481934606d8b1c710c724f3612a40bc2bf2011a353c88\": container with ID starting with 24414dbda1894e4effc481934606d8b1c710c724f3612a40bc2bf2011a353c88 not found: ID does not exist"
Oct 30 19:15:44.841554 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.841476    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"]
Oct 30 19:15:44.844470 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.844439    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f5763a-669585666c-7dprk"]
Oct 30 19:15:44.950024 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.949990    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"]
Oct 30 19:15:44.955463 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:44.955431    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f5763a-6fc97d7d8c-7wdtg"]
Oct 30 19:15:45.477850 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:45.477815    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee7e187-7e2a-4105-a4d0-1fe11a63464f" path="/var/lib/kubelet/pods/7ee7e187-7e2a-4105-a4d0-1fe11a63464f/volumes"
Oct 30 19:15:45.478352 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:45.478337    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe3f0683-26aa-408e-98d2-97d34bba0d14" path="/var/lib/kubelet/pods/fe3f0683-26aa-408e-98d2-97d34bba0d14/volumes"
Oct 30 19:15:51.391361 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:51.391325    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:15:53.272233 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:53.272191    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:15:55.882553 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:55.882513    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"]
Oct 30 19:15:55.883226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:55.882715    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7" podUID="6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" containerName="discovery" containerID="cri-o://aeed9f033663a4831e9c121022c3946ce5fdada900889bf1469d894786e4ee6c" gracePeriod=30
Oct 30 19:15:56.006631 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.006601    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"]
Oct 30 19:15:56.006969 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.006940    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy" containerID="cri-o://758c22e6d437c2ab93c918557e173533f3862b78d65b86536582211825979420" gracePeriod=30
Oct 30 19:15:56.218052 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.218018    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:15:56.328689 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.328651    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-local-certs\") pod \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") "
Oct 30 19:15:56.328898 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.328700    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-cacerts\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-cacerts\") pod \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") "
Oct 30 19:15:56.328898 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.328736    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-kubeconfig\") pod \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") "
Oct 30 19:15:56.328898 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.328789    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-kube-api-access-ltn5z\" (UniqueName: \"kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-kube-api-access-ltn5z\") pod \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") "
Oct 30 19:15:56.328898 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.328891    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-token\" (UniqueName: \"kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-token\") pod \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") "
Oct 30 19:15:56.329209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.328992    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-dns-cert\") pod \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") "
Oct 30 19:15:56.329209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.329090    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-ca-configmap\") pod \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\" (UID: \"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279\") "
Oct 30 19:15:56.329751 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.329710    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-ca-configmap" pod "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" (UID: "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:56.331833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.331802    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-local-certs" pod "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" (UID: "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:56.331833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.331816    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-cacerts" pod "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" (UID: "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:56.331959 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.331830    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-kube-api-access-ltn5z" pod "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" (UID: "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279"). InnerVolumeSpecName "kube-api-access-ltn5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:56.331959 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.331871    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-dns-cert" pod "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" (UID: "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:56.332279 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.332263    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-token" pod "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" (UID: "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:56.332570 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.332548    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-kubeconfig" pod "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" (UID: "6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:56.431310 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.431227    6014 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-ca-configmap\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:56.431310 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.431258    6014 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-local-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:56.431310 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.431269    6014 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-cacerts\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:56.431310 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.431278    6014 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-kubeconfig\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:56.431310 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.431286    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltn5z\" (UniqueName: \"kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-kube-api-access-ltn5z\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:56.431310 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.431296    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:56.431310 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.431305    6014 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279-istio-csr-dns-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:56.776469 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.776383    6014 generic.go:356] "Generic (PLEG): container finished" podID="6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" containerID="aeed9f033663a4831e9c121022c3946ce5fdada900889bf1469d894786e4ee6c" exitCode=0
Oct 30 19:15:56.776610 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.776491    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"
Oct 30 19:15:56.776610 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.776532    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7" event={"ID":"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279","Type":"ContainerDied","Data":"aeed9f033663a4831e9c121022c3946ce5fdada900889bf1469d894786e4ee6c"}
Oct 30 19:15:56.776610 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.776586    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7" event={"ID":"6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279","Type":"ContainerDied","Data":"5baa65bc454555369ec55171ed440616da88e731731083588b789257dc1c769d"}
Oct 30 19:15:56.776747 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.776612    6014 scope.go:122] "RemoveContainer" containerID="aeed9f033663a4831e9c121022c3946ce5fdada900889bf1469d894786e4ee6c"
Oct 30 19:15:56.788051 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.788029    6014 scope.go:122] "RemoveContainer" containerID="aeed9f033663a4831e9c121022c3946ce5fdada900889bf1469d894786e4ee6c"
Oct 30 19:15:56.788332 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:15:56.788315    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeed9f033663a4831e9c121022c3946ce5fdada900889bf1469d894786e4ee6c\": container with ID starting with aeed9f033663a4831e9c121022c3946ce5fdada900889bf1469d894786e4ee6c not found: ID does not exist" containerID="aeed9f033663a4831e9c121022c3946ce5fdada900889bf1469d894786e4ee6c"
Oct 30 19:15:56.788397 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.788340    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeed9f033663a4831e9c121022c3946ce5fdada900889bf1469d894786e4ee6c"} err="failed to get container status \"aeed9f033663a4831e9c121022c3946ce5fdada900889bf1469d894786e4ee6c\": rpc error: code = NotFound desc = could not find container \"aeed9f033663a4831e9c121022c3946ce5fdada900889bf1469d894786e4ee6c\": container with ID starting with aeed9f033663a4831e9c121022c3946ce5fdada900889bf1469d894786e4ee6c not found: ID does not exist"
Oct 30 19:15:56.950253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.950222    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"]
Oct 30 19:15:56.955628 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:56.955597    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bgxf7"]
Oct 30 19:15:57.489280 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:15:57.489247    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" path="/var/lib/kubelet/pods/6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279/volumes"
Oct 30 19:16:01.315434 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.315399    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:16:01.403362 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.403332    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-envoy\") pod \"c565e581-0906-402f-8dbe-16154942ba9e\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") "
Oct 30 19:16:01.403362 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.403367    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c565e581-0906-402f-8dbe-16154942ba9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c565e581-0906-402f-8dbe-16154942ba9e-istiod-ca-cert\") pod \"c565e581-0906-402f-8dbe-16154942ba9e\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") "
Oct 30 19:16:01.403648 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.403415    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-istio-token\" (UniqueName: \"kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-istio-token\") pod \"c565e581-0906-402f-8dbe-16154942ba9e\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") "
Oct 30 19:16:01.403648 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.403504    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-credential-socket\") pod \"c565e581-0906-402f-8dbe-16154942ba9e\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") "
Oct 30 19:16:01.403648 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.403632    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-socket\") pod \"c565e581-0906-402f-8dbe-16154942ba9e\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") "
Oct 30 19:16:01.403815 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.403672    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-certs\") pod \"c565e581-0906-402f-8dbe-16154942ba9e\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") "
Oct 30 19:16:01.403815 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.403725    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c565e581-0906-402f-8dbe-16154942ba9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c565e581-0906-402f-8dbe-16154942ba9e-istio-podinfo\") pod \"c565e581-0906-402f-8dbe-16154942ba9e\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") "
Oct 30 19:16:01.403927 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.403833    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-data\") pod \"c565e581-0906-402f-8dbe-16154942ba9e\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") "
Oct 30 19:16:01.403927 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.403861    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-kube-api-access-qd6qt\" (UniqueName: \"kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-kube-api-access-qd6qt\") pod \"c565e581-0906-402f-8dbe-16154942ba9e\" (UID: \"c565e581-0906-402f-8dbe-16154942ba9e\") "
Oct 30 19:16:01.404038 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.403947    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-credential-socket" pod "c565e581-0906-402f-8dbe-16154942ba9e" (UID: "c565e581-0906-402f-8dbe-16154942ba9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.404479 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.404105    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-socket" pod "c565e581-0906-402f-8dbe-16154942ba9e" (UID: "c565e581-0906-402f-8dbe-16154942ba9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.404682 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.404493    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c565e581-0906-402f-8dbe-16154942ba9e-istiod-ca-cert" pod "c565e581-0906-402f-8dbe-16154942ba9e" (UID: "c565e581-0906-402f-8dbe-16154942ba9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:01.404784 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.404731    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-certs" pod "c565e581-0906-402f-8dbe-16154942ba9e" (UID: "c565e581-0906-402f-8dbe-16154942ba9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.405043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.404974    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c565e581-0906-402f-8dbe-16154942ba9e-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.405043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.405000    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.405043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.405014    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.405474 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.405446    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-data" pod "c565e581-0906-402f-8dbe-16154942ba9e" (UID: "c565e581-0906-402f-8dbe-16154942ba9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.406783 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.406748    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-istio-token" pod "c565e581-0906-402f-8dbe-16154942ba9e" (UID: "c565e581-0906-402f-8dbe-16154942ba9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:01.407568 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.407544    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-envoy" pod "c565e581-0906-402f-8dbe-16154942ba9e" (UID: "c565e581-0906-402f-8dbe-16154942ba9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.408554 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.408517    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-kube-api-access-qd6qt" pod "c565e581-0906-402f-8dbe-16154942ba9e" (UID: "c565e581-0906-402f-8dbe-16154942ba9e"). InnerVolumeSpecName "kube-api-access-qd6qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:01.408554 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.408542    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c565e581-0906-402f-8dbe-16154942ba9e-istio-podinfo" pod "c565e581-0906-402f-8dbe-16154942ba9e" (UID: "c565e581-0906-402f-8dbe-16154942ba9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:01.507089 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.506417    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.507089 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.506444    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd6qt\" (UniqueName: \"kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-kube-api-access-qd6qt\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.507089 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.506455    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.507089 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.506469    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c565e581-0906-402f-8dbe-16154942ba9e-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.507089 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.506479    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c565e581-0906-402f-8dbe-16154942ba9e-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.507089 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.506502    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c565e581-0906-402f-8dbe-16154942ba9e-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.828119 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.828082    6014 generic.go:356] "Generic (PLEG): container finished" podID="c565e581-0906-402f-8dbe-16154942ba9e" containerID="758c22e6d437c2ab93c918557e173533f3862b78d65b86536582211825979420" exitCode=0
Oct 30 19:16:01.828356 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.828178    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"
Oct 30 19:16:01.828356 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.828173    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" event={"ID":"c565e581-0906-402f-8dbe-16154942ba9e","Type":"ContainerDied","Data":"758c22e6d437c2ab93c918557e173533f3862b78d65b86536582211825979420"}
Oct 30 19:16:01.828356 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.828277    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k" event={"ID":"c565e581-0906-402f-8dbe-16154942ba9e","Type":"ContainerDied","Data":"882bcbe54bdaf7100f08e2576499c77a6b11d896633113d361d4f5ff635c2f3b"}
Oct 30 19:16:01.828356 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.828308    6014 scope.go:122] "RemoveContainer" containerID="758c22e6d437c2ab93c918557e173533f3862b78d65b86536582211825979420"
Oct 30 19:16:01.839969 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.839946    6014 scope.go:122] "RemoveContainer" containerID="758c22e6d437c2ab93c918557e173533f3862b78d65b86536582211825979420"
Oct 30 19:16:01.840357 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:16:01.840335    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758c22e6d437c2ab93c918557e173533f3862b78d65b86536582211825979420\": container with ID starting with 758c22e6d437c2ab93c918557e173533f3862b78d65b86536582211825979420 not found: ID does not exist" containerID="758c22e6d437c2ab93c918557e173533f3862b78d65b86536582211825979420"
Oct 30 19:16:01.840429 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:01.840369    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758c22e6d437c2ab93c918557e173533f3862b78d65b86536582211825979420"} err="failed to get container status \"758c22e6d437c2ab93c918557e173533f3862b78d65b86536582211825979420\": rpc error: code = NotFound desc = could not find container \"758c22e6d437c2ab93c918557e173533f3862b78d65b86536582211825979420\": container with ID starting with 758c22e6d437c2ab93c918557e173533f3862b78d65b86536582211825979420 not found: ID does not exist"
Oct 30 19:16:02.033192 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:02.032196    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"]
Oct 30 19:16:02.056161 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:02.055628    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-shx9k"]
Oct 30 19:16:03.477204 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:03.477130    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c565e581-0906-402f-8dbe-16154942ba9e" path="/var/lib/kubelet/pods/c565e581-0906-402f-8dbe-16154942ba9e/volumes"
Oct 30 19:16:09.085329 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.085288    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"]
Oct 30 19:16:09.086182 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.086111    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9" podUID="b85e7d51-316a-4e68-87e0-03217deade17" containerName="maas-api" containerID="cri-o://c7876f441be1f72c928b416632c3e4054317349119f70445d2126507ec460bd3" gracePeriod=30
Oct 30 19:16:09.384095 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.384067    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:16:09.557624 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.557592    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b85e7d51-316a-4e68-87e0-03217deade17-kube-api-access-dldxv\" (UniqueName: \"kubernetes.io/projected/b85e7d51-316a-4e68-87e0-03217deade17-kube-api-access-dldxv\") pod \"b85e7d51-316a-4e68-87e0-03217deade17\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") "
Oct 30 19:16:09.557803 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.557647    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-metrics-certs\") pod \"b85e7d51-316a-4e68-87e0-03217deade17\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") "
Oct 30 19:16:09.557803 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.557697    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-maas-api-tls\") pod \"b85e7d51-316a-4e68-87e0-03217deade17\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") "
Oct 30 19:16:09.557803 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.557748    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b85e7d51-316a-4e68-87e0-03217deade17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85e7d51-316a-4e68-87e0-03217deade17-openshift-service-ca\") pod \"b85e7d51-316a-4e68-87e0-03217deade17\" (UID: \"b85e7d51-316a-4e68-87e0-03217deade17\") "
Oct 30 19:16:09.558153 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.558115    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85e7d51-316a-4e68-87e0-03217deade17-openshift-service-ca" pod "b85e7d51-316a-4e68-87e0-03217deade17" (UID: "b85e7d51-316a-4e68-87e0-03217deade17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:09.558547 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.558481    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85e7d51-316a-4e68-87e0-03217deade17-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:09.560156 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.560114    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-maas-api-tls" pod "b85e7d51-316a-4e68-87e0-03217deade17" (UID: "b85e7d51-316a-4e68-87e0-03217deade17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:09.560263 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.560175    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-metrics-certs" pod "b85e7d51-316a-4e68-87e0-03217deade17" (UID: "b85e7d51-316a-4e68-87e0-03217deade17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:09.560263 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.560210    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85e7d51-316a-4e68-87e0-03217deade17-kube-api-access-dldxv" pod "b85e7d51-316a-4e68-87e0-03217deade17" (UID: "b85e7d51-316a-4e68-87e0-03217deade17"). InnerVolumeSpecName "kube-api-access-dldxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:09.660561 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.660470    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dldxv\" (UniqueName: \"kubernetes.io/projected/b85e7d51-316a-4e68-87e0-03217deade17-kube-api-access-dldxv\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:09.660561 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.660504    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:09.660561 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.660517    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85e7d51-316a-4e68-87e0-03217deade17-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:09.896474 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.896443    6014 generic.go:356] "Generic (PLEG): container finished" podID="b85e7d51-316a-4e68-87e0-03217deade17" containerID="c7876f441be1f72c928b416632c3e4054317349119f70445d2126507ec460bd3" exitCode=0
Oct 30 19:16:09.896673 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.896494    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"
Oct 30 19:16:09.896673 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.896519    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9" event={"ID":"b85e7d51-316a-4e68-87e0-03217deade17","Type":"ContainerDied","Data":"c7876f441be1f72c928b416632c3e4054317349119f70445d2126507ec460bd3"}
Oct 30 19:16:09.896673 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.896581    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9" event={"ID":"b85e7d51-316a-4e68-87e0-03217deade17","Type":"ContainerDied","Data":"cf1ac3eedb59c58bc3982371782fe5de4c5a264aadfc57d7e0d7f0ad7ed7c207"}
Oct 30 19:16:09.896673 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.896611    6014 scope.go:122] "RemoveContainer" containerID="c7876f441be1f72c928b416632c3e4054317349119f70445d2126507ec460bd3"
Oct 30 19:16:09.909054 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.909037    6014 scope.go:122] "RemoveContainer" containerID="c7876f441be1f72c928b416632c3e4054317349119f70445d2126507ec460bd3"
Oct 30 19:16:09.909469 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:16:09.909450    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7876f441be1f72c928b416632c3e4054317349119f70445d2126507ec460bd3\": container with ID starting with c7876f441be1f72c928b416632c3e4054317349119f70445d2126507ec460bd3 not found: ID does not exist" containerID="c7876f441be1f72c928b416632c3e4054317349119f70445d2126507ec460bd3"
Oct 30 19:16:09.909574 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:09.909481    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7876f441be1f72c928b416632c3e4054317349119f70445d2126507ec460bd3"} err="failed to get container status \"c7876f441be1f72c928b416632c3e4054317349119f70445d2126507ec460bd3\": rpc error: code = NotFound desc = could not find container \"c7876f441be1f72c928b416632c3e4054317349119f70445d2126507ec460bd3\": container with ID starting with c7876f441be1f72c928b416632c3e4054317349119f70445d2126507ec460bd3 not found: ID does not exist"
Oct 30 19:16:10.063795 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:10.063762    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"]
Oct 30 19:16:10.068590 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:10.068565    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ebeba-6f76dc64f9-rmlr9"]
Oct 30 19:16:11.478740 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:11.478710    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b85e7d51-316a-4e68-87e0-03217deade17" path="/var/lib/kubelet/pods/b85e7d51-316a-4e68-87e0-03217deade17/volumes"
Oct 30 19:16:16.456227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.456183    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9104-54746b"]
Oct 30 19:16:16.456908 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.456893    6014 cpu_manager.go:414] "containerMap: removing container" podUID="b85e7d51-316a-4e68-87e0-03217deade17" containerName="maas-api"
Oct 30 19:16:16.456974 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.456909    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b85e7d51-316a-4e68-87e0-03217deade17" containerName="maas-api"
Oct 30 19:16:16.456974 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.456925    6014 cpu_manager.go:414] "containerMap: removing container" podUID="fe3f0683-26aa-408e-98d2-97d34bba0d14" containerName="payload-processing"
Oct 30 19:16:16.456974 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.456930    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3f0683-26aa-408e-98d2-97d34bba0d14" containerName="payload-processing"
Oct 30 19:16:16.456974 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.456938    6014 cpu_manager.go:414] "containerMap: removing container" podUID="6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" containerName="discovery"
Oct 30 19:16:16.456974 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.456943    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" containerName="discovery"
Oct 30 19:16:16.456974 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.456967    6014 cpu_manager.go:414] "containerMap: removing container" podUID="7ee7e187-7e2a-4105-a4d0-1fe11a63464f" containerName="payload-pre-processing"
Oct 30 19:16:16.456974 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.456973    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee7e187-7e2a-4105-a4d0-1fe11a63464f" containerName="payload-pre-processing"
Oct 30 19:16:16.457209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.456995    6014 cpu_manager.go:414] "containerMap: removing container" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy"
Oct 30 19:16:16.457209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.457000    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy"
Oct 30 19:16:16.457209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.457110    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3f0683-26aa-408e-98d2-97d34bba0d14" containerName="payload-processing"
Oct 30 19:16:16.457209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.457118    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a26ab8b-b868-4e14-bf8b-6ba8cf4a3279" containerName="discovery"
Oct 30 19:16:16.457209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.457127    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee7e187-7e2a-4105-a4d0-1fe11a63464f" containerName="payload-pre-processing"
Oct 30 19:16:16.457209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.457152    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="c565e581-0906-402f-8dbe-16154942ba9e" containerName="istio-proxy"
Oct 30 19:16:16.457209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.457163    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b85e7d51-316a-4e68-87e0-03217deade17" containerName="maas-api"
Oct 30 19:16:16.460877 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.460852    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9104-54746b"
Oct 30 19:16:16.495510 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.495421    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdv2b\" (UniqueName: \"kubernetes.io/projected/3bba7da7-4910-4ed4-bc79-dc751d08b640-kube-api-access-kdv2b\") pod \"test-curl-9104-54746b\" (UID: \"3bba7da7-4910-4ed4-bc79-dc751d08b640\") " pod="opendatahub/test-curl-9104-54746b"
Oct 30 19:16:16.519255 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.519216    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9104-54746b"]
Oct 30 19:16:16.564542 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.564515    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7v9w4\""
Oct 30 19:16:16.597521 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.597483    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdv2b\" (UniqueName: \"kubernetes.io/projected/3bba7da7-4910-4ed4-bc79-dc751d08b640-kube-api-access-kdv2b\") pod \"test-curl-9104-54746b\" (UID: \"3bba7da7-4910-4ed4-bc79-dc751d08b640\") " pod="opendatahub/test-curl-9104-54746b"
Oct 30 19:16:16.655932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.655888    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdv2b\" (UniqueName: \"kubernetes.io/projected/3bba7da7-4910-4ed4-bc79-dc751d08b640-kube-api-access-kdv2b\") pod \"test-curl-9104-54746b\" (UID: \"3bba7da7-4910-4ed4-bc79-dc751d08b640\") " pod="opendatahub/test-curl-9104-54746b"
Oct 30 19:16:16.807191 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:16.807066    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9104-54746b"
Oct 30 19:16:17.180900 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:17.180869    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9104-54746b"]
Oct 30 19:16:17.182881 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:16:17.182854    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bba7da7_4910_4ed4_bc79_dc751d08b640.slice/crio-babee612bf13afc18b5220fc0d11d0d249d19fb1431954267bccb2051a0cdbd1 WatchSource:0}: Error finding container babee612bf13afc18b5220fc0d11d0d249d19fb1431954267bccb2051a0cdbd1: Status 404 returned error can't find the container with id babee612bf13afc18b5220fc0d11d0d249d19fb1431954267bccb2051a0cdbd1
Oct 30 19:16:17.958831 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:17.958796    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9104-54746b" event={"ID":"3bba7da7-4910-4ed4-bc79-dc751d08b640","Type":"ContainerStarted","Data":"babee612bf13afc18b5220fc0d11d0d249d19fb1431954267bccb2051a0cdbd1"}
Oct 30 19:16:18.967372 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:18.967319    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9104-54746b" event={"ID":"3bba7da7-4910-4ed4-bc79-dc751d08b640","Type":"ContainerStarted","Data":"f04f82033ab7eeaf14d8f89833e41f48d461cdb6cb88dcd92d0a0c67836bb055"}
Oct 30 19:16:19.081306 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:19.081254    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9104-54746b" podStartSLOduration=1.756351754 podStartE2EDuration="3.081237743s" totalImagesPullingTime="1.324885989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:19.078542055 +0000 UTC m=+2217.721709670" watchObservedRunningTime="2026-10-30 19:16:19.081237743 +0000 UTC m=+2217.724405358"
Oct 30 19:16:20.571371 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:20.571335    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9104-54746b"]
Oct 30 19:16:20.574620 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:20.574591    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9104-54746b"]
Oct 30 19:16:20.982547 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:20.982423    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9104-54746b" podUID="3bba7da7-4910-4ed4-bc79-dc751d08b640" containerName="test-curl-9104-54746b" containerID="cri-o://f04f82033ab7eeaf14d8f89833e41f48d461cdb6cb88dcd92d0a0c67836bb055" gracePeriod=2
Oct 30 19:16:21.033281 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:16:21.033235    6014 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9104-54746b\" is forbidden: User \"system:node:ip-10-0-143-97.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-97.us-west-1.compute.internal' and this object" podUID="3bba7da7-4910-4ed4-bc79-dc751d08b640" pod="opendatahub/test-curl-9104-54746b"
Oct 30 19:16:21.535347 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:16:21.535303    6014 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9104-54746b\" is forbidden: User \"system:node:ip-10-0-143-97.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-97.us-west-1.compute.internal' and this object" podUID="3bba7da7-4910-4ed4-bc79-dc751d08b640" pod="opendatahub/test-curl-9104-54746b"
Oct 30 19:16:23.232543 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:23.232508    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9104-54746b"
Oct 30 19:16:23.259078 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:16:23.258996    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:16:23.311446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:23.311405    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bba7da7-4910-4ed4-bc79-dc751d08b640-kube-api-access-kdv2b\" (UniqueName: \"kubernetes.io/projected/3bba7da7-4910-4ed4-bc79-dc751d08b640-kube-api-access-kdv2b\") pod \"3bba7da7-4910-4ed4-bc79-dc751d08b640\" (UID: \"3bba7da7-4910-4ed4-bc79-dc751d08b640\") "
Oct 30 19:16:23.313686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:23.313656    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bba7da7-4910-4ed4-bc79-dc751d08b640-kube-api-access-kdv2b" pod "3bba7da7-4910-4ed4-bc79-dc751d08b640" (UID: "3bba7da7-4910-4ed4-bc79-dc751d08b640"). InnerVolumeSpecName "kube-api-access-kdv2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:23.413253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:23.413214    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdv2b\" (UniqueName: \"kubernetes.io/projected/3bba7da7-4910-4ed4-bc79-dc751d08b640-kube-api-access-kdv2b\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.478114 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:23.478084    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bba7da7-4910-4ed4-bc79-dc751d08b640" path="/var/lib/kubelet/pods/3bba7da7-4910-4ed4-bc79-dc751d08b640/volumes"
Oct 30 19:16:24.004187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:24.004152    6014 generic.go:356] "Generic (PLEG): container finished" podID="3bba7da7-4910-4ed4-bc79-dc751d08b640" containerID="f04f82033ab7eeaf14d8f89833e41f48d461cdb6cb88dcd92d0a0c67836bb055" exitCode=137
Oct 30 19:16:24.004401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:24.004222    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9104-54746b"
Oct 30 19:16:24.004401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:24.004255    6014 scope.go:122] "RemoveContainer" containerID="f04f82033ab7eeaf14d8f89833e41f48d461cdb6cb88dcd92d0a0c67836bb055"
Oct 30 19:16:24.016275 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:24.016255    6014 scope.go:122] "RemoveContainer" containerID="f04f82033ab7eeaf14d8f89833e41f48d461cdb6cb88dcd92d0a0c67836bb055"
Oct 30 19:16:24.016737 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:16:24.016718    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04f82033ab7eeaf14d8f89833e41f48d461cdb6cb88dcd92d0a0c67836bb055\": container with ID starting with f04f82033ab7eeaf14d8f89833e41f48d461cdb6cb88dcd92d0a0c67836bb055 not found: ID does not exist" containerID="f04f82033ab7eeaf14d8f89833e41f48d461cdb6cb88dcd92d0a0c67836bb055"
Oct 30 19:16:24.016789 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:24.016745    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04f82033ab7eeaf14d8f89833e41f48d461cdb6cb88dcd92d0a0c67836bb055"} err="failed to get container status \"f04f82033ab7eeaf14d8f89833e41f48d461cdb6cb88dcd92d0a0c67836bb055\": rpc error: code = NotFound desc = could not find container \"f04f82033ab7eeaf14d8f89833e41f48d461cdb6cb88dcd92d0a0c67836bb055\": container with ID starting with f04f82033ab7eeaf14d8f89833e41f48d461cdb6cb88dcd92d0a0c67836bb055 not found: ID does not exist"
Oct 30 19:16:28.442802 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.442765    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"]
Oct 30 19:16:28.443491 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.443475    6014 cpu_manager.go:414] "containerMap: removing container" podUID="3bba7da7-4910-4ed4-bc79-dc751d08b640" containerName="test-curl-9104-54746b"
Oct 30 19:16:28.443543 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.443492    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bba7da7-4910-4ed4-bc79-dc751d08b640" containerName="test-curl-9104-54746b"
Oct 30 19:16:28.443647 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.443637    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bba7da7-4910-4ed4-bc79-dc751d08b640" containerName="test-curl-9104-54746b"
Oct 30 19:16:28.449064 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.449044    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.504194 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.501121    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-714b2c-openshift-default-dockercfg-8bwvq\""
Oct 30 19:16:28.553763 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.553727    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"]
Oct 30 19:16:28.596587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.596551    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-certs\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.601499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.601465    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749b5\" (UniqueName: \"kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-kube-api-access-749b5\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.606202 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.606111    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-token\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.610737 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.610698    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-socket\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.611102 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.611072    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-data\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.618619 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.618576    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-credential-socket\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.619947 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.619908    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-envoy\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.623197 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.621036    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2baa7c-faac-4276-b89e-2b16917d3495-istiod-ca-cert\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.623472 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.623447    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-podinfo\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.725362 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.725270    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-envoy\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.725362 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.725328    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2baa7c-faac-4276-b89e-2b16917d3495-istiod-ca-cert\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.725630 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.725368    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-podinfo\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.725630 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.725431    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-certs\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.725630 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.725552    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749b5\" (UniqueName: \"kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-kube-api-access-749b5\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.725630 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.725579    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-token\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.725630 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.725624    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-socket\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.725916 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.725684    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-data\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.725916 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.725728    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-credential-socket\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.725916 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.725893    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-certs\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.726096 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.726058    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-credential-socket\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.726213 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.726168    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-socket\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.726297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.726272    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2baa7c-faac-4276-b89e-2b16917d3495-istiod-ca-cert\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.726376 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.726353    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-data\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.728087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.728067    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-podinfo\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.728421 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.728399    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-envoy\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.783585 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.783551    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-token\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:28.783756 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:28.783680    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-749b5\" (UniqueName: \"kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-kube-api-access-749b5\") pod \"e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") " pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:29.063603 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:29.063562    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:29.497013 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:29.496989    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"]
Oct 30 19:16:29.499078 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:16:29.499049    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2baa7c_faac_4276_b89e_2b16917d3495.slice/crio-68a1eede3e24b934927629bd4c389c758fa3719050481c61cc90f0498b31400c WatchSource:0}: Error finding container 68a1eede3e24b934927629bd4c389c758fa3719050481c61cc90f0498b31400c: Status 404 returned error can't find the container with id 68a1eede3e24b934927629bd4c389c758fa3719050481c61cc90f0498b31400c
Oct 30 19:16:29.501812 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:29.501780    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:16:29.501902 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:29.501852    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:16:29.501948 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:29.501900    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:16:30.048733 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:30.048685    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9" event={"ID":"cd2baa7c-faac-4276-b89e-2b16917d3495","Type":"ContainerStarted","Data":"253907f23b82ed9f9d85af5cc97549680fdbaf6b8b9d5fde2abfd5bb23ff6bf7"}
Oct 30 19:16:30.048934 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:30.048787    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9" event={"ID":"cd2baa7c-faac-4276-b89e-2b16917d3495","Type":"ContainerStarted","Data":"68a1eede3e24b934927629bd4c389c758fa3719050481c61cc90f0498b31400c"}
Oct 30 19:16:30.064222 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:30.064189    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:30.070415 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:30.070387    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:30.166203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:30.166122    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9" podStartSLOduration=2.166108312 podStartE2EDuration="2.166108312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:30.16581398 +0000 UTC m=+2228.808981593" watchObservedRunningTime="2026-10-30 19:16:30.166108312 +0000 UTC m=+2228.809275926"
Oct 30 19:16:31.056282 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:31.056249    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:31.057193 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:31.057169    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:16:32.780811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:32.780771    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"]
Oct 30 19:16:32.896288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:32.896250    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"]
Oct 30 19:16:37.105528 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.105491    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"]
Oct 30 19:16:37.110267 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.110246    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"
Oct 30 19:16:37.195671 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.195640    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cedbbcda-3367-4df0-bbca-e9af8db159cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm\" (UID: \"cedbbcda-3367-4df0-bbca-e9af8db159cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"
Oct 30 19:16:37.195857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.195785    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp65l\" (UniqueName: \"kubernetes.io/projected/cedbbcda-3367-4df0-bbca-e9af8db159cd-kube-api-access-cp65l\") pod \"payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm\" (UID: \"cedbbcda-3367-4df0-bbca-e9af8db159cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"
Oct 30 19:16:37.264174 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.262945    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-714b2c-dockercfg-8p95s\""
Oct 30 19:16:37.264174 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.263023    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-714b2c\""
Oct 30 19:16:37.293907 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.293872    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"]
Oct 30 19:16:37.298309 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.298277    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cedbbcda-3367-4df0-bbca-e9af8db159cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm\" (UID: \"cedbbcda-3367-4df0-bbca-e9af8db159cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"
Oct 30 19:16:37.299549 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.299522    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp65l\" (UniqueName: \"kubernetes.io/projected/cedbbcda-3367-4df0-bbca-e9af8db159cd-kube-api-access-cp65l\") pod \"payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm\" (UID: \"cedbbcda-3367-4df0-bbca-e9af8db159cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"
Oct 30 19:16:37.300226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.299375    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cedbbcda-3367-4df0-bbca-e9af8db159cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm\" (UID: \"cedbbcda-3367-4df0-bbca-e9af8db159cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"
Oct 30 19:16:37.304197 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.304177    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"]
Oct 30 19:16:37.317798 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.317769    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"
Oct 30 19:16:37.366203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.349933    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"]
Oct 30 19:16:37.366203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.360577    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp65l\" (UniqueName: \"kubernetes.io/projected/cedbbcda-3367-4df0-bbca-e9af8db159cd-kube-api-access-cp65l\") pod \"payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm\" (UID: \"cedbbcda-3367-4df0-bbca-e9af8db159cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"
Oct 30 19:16:37.401585 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.401494    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlzrm\" (UniqueName: \"kubernetes.io/projected/b420a535-af91-4068-ae30-fef541da631b-kube-api-access-zlzrm\") pod \"payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk\" (UID: \"b420a535-af91-4068-ae30-fef541da631b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"
Oct 30 19:16:37.403225 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.401860    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b420a535-af91-4068-ae30-fef541da631b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk\" (UID: \"b420a535-af91-4068-ae30-fef541da631b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"
Oct 30 19:16:37.425047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.425015    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"
Oct 30 19:16:37.504773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.503418    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b420a535-af91-4068-ae30-fef541da631b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk\" (UID: \"b420a535-af91-4068-ae30-fef541da631b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"
Oct 30 19:16:37.504773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.503631    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzrm\" (UniqueName: \"kubernetes.io/projected/b420a535-af91-4068-ae30-fef541da631b-kube-api-access-zlzrm\") pod \"payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk\" (UID: \"b420a535-af91-4068-ae30-fef541da631b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"
Oct 30 19:16:37.509964 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.505129    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b420a535-af91-4068-ae30-fef541da631b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk\" (UID: \"b420a535-af91-4068-ae30-fef541da631b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"
Oct 30 19:16:37.560947 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.560913    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlzrm\" (UniqueName: \"kubernetes.io/projected/b420a535-af91-4068-ae30-fef541da631b-kube-api-access-zlzrm\") pod \"payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk\" (UID: \"b420a535-af91-4068-ae30-fef541da631b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"
Oct 30 19:16:37.688391 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.688301    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"
Oct 30 19:16:37.782070 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.782017    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9" podUID="97b68589-f0f6-4d49-b519-9d30746433ca" containerName="payload-processing" containerID="cri-o://8a404236ae55b1b8ac99a51fd7e7315ffa5d364d5709cbac5896816ecee9cffc" gracePeriod=25
Oct 30 19:16:37.817857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.817821    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"]
Oct 30 19:16:37.835055 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:16:37.835022    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcedbbcda_3367_4df0_bbca_e9af8db159cd.slice/crio-c07957956c8549a285f33dca0a6711cc123093e8be507b33405edb689f84b099 WatchSource:0}: Error finding container c07957956c8549a285f33dca0a6711cc123093e8be507b33405edb689f84b099: Status 404 returned error can't find the container with id c07957956c8549a285f33dca0a6711cc123093e8be507b33405edb689f84b099
Oct 30 19:16:37.897591 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:37.897536    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6" podUID="ea5eebbf-70d4-48fe-827a-965f191a8938" containerName="payload-pre-processing" containerID="cri-o://7a9be7426a674d37965faca4f3cb50e9036373362f4fe5122f6bbdb5937ad8fc" gracePeriod=25
Oct 30 19:16:38.061720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.061694    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"
Oct 30 19:16:38.075873 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.075841    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"]
Oct 30 19:16:38.076034 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:16:38.076010    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb420a535_af91_4068_ae30_fef541da631b.slice/crio-1d2749f61f17742d908d0733ff3f422a71bc669b1cda4655b369252d40d3c8e9 WatchSource:0}: Error finding container 1d2749f61f17742d908d0733ff3f422a71bc669b1cda4655b369252d40d3c8e9: Status 404 returned error can't find the container with id 1d2749f61f17742d908d0733ff3f422a71bc669b1cda4655b369252d40d3c8e9
Oct 30 19:16:38.116927 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.116894    6014 generic.go:356] "Generic (PLEG): container finished" podID="97b68589-f0f6-4d49-b519-9d30746433ca" containerID="8a404236ae55b1b8ac99a51fd7e7315ffa5d364d5709cbac5896816ecee9cffc" exitCode=0
Oct 30 19:16:38.117399 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.116953    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"
Oct 30 19:16:38.117399 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.116985    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9" event={"ID":"97b68589-f0f6-4d49-b519-9d30746433ca","Type":"ContainerDied","Data":"8a404236ae55b1b8ac99a51fd7e7315ffa5d364d5709cbac5896816ecee9cffc"}
Oct 30 19:16:38.117399 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.117042    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9" event={"ID":"97b68589-f0f6-4d49-b519-9d30746433ca","Type":"ContainerDied","Data":"bd814c74cef4053e017e27bf9f045fd2271515b4d17f0c41007d4b699465d34c"}
Oct 30 19:16:38.117399 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.117074    6014 scope.go:122] "RemoveContainer" containerID="8a404236ae55b1b8ac99a51fd7e7315ffa5d364d5709cbac5896816ecee9cffc"
Oct 30 19:16:38.119696 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.119670    6014 generic.go:356] "Generic (PLEG): container finished" podID="ea5eebbf-70d4-48fe-827a-965f191a8938" containerID="7a9be7426a674d37965faca4f3cb50e9036373362f4fe5122f6bbdb5937ad8fc" exitCode=0
Oct 30 19:16:38.119814 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.119759    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6" event={"ID":"ea5eebbf-70d4-48fe-827a-965f191a8938","Type":"ContainerDied","Data":"7a9be7426a674d37965faca4f3cb50e9036373362f4fe5122f6bbdb5937ad8fc"}
Oct 30 19:16:38.121629 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.121596    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk" event={"ID":"b420a535-af91-4068-ae30-fef541da631b","Type":"ContainerStarted","Data":"1d2749f61f17742d908d0733ff3f422a71bc669b1cda4655b369252d40d3c8e9"}
Oct 30 19:16:38.127219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.125597    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b68589-f0f6-4d49-b519-9d30746433ca-kube-api-access-5t574\" (UniqueName: \"kubernetes.io/projected/97b68589-f0f6-4d49-b519-9d30746433ca-kube-api-access-5t574\") pod \"97b68589-f0f6-4d49-b519-9d30746433ca\" (UID: \"97b68589-f0f6-4d49-b519-9d30746433ca\") "
Oct 30 19:16:38.127219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.125952    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b68589-f0f6-4d49-b519-9d30746433ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b68589-f0f6-4d49-b519-9d30746433ca-plugins-config-volume\") pod \"97b68589-f0f6-4d49-b519-9d30746433ca\" (UID: \"97b68589-f0f6-4d49-b519-9d30746433ca\") "
Oct 30 19:16:38.127219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.126857    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b68589-f0f6-4d49-b519-9d30746433ca-plugins-config-volume" pod "97b68589-f0f6-4d49-b519-9d30746433ca" (UID: "97b68589-f0f6-4d49-b519-9d30746433ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:38.127739 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.127714    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm" event={"ID":"cedbbcda-3367-4df0-bbca-e9af8db159cd","Type":"ContainerStarted","Data":"0bc8a1e0da64b982aa04f7457f1ce49cc2b7b44b853e800c15328519c93f01de"}
Oct 30 19:16:38.128035 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.128016    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm" event={"ID":"cedbbcda-3367-4df0-bbca-e9af8db159cd","Type":"ContainerStarted","Data":"c07957956c8549a285f33dca0a6711cc123093e8be507b33405edb689f84b099"}
Oct 30 19:16:38.128414 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.128377    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"
Oct 30 19:16:38.128556 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.128292    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b68589-f0f6-4d49-b519-9d30746433ca-kube-api-access-5t574" pod "97b68589-f0f6-4d49-b519-9d30746433ca" (UID: "97b68589-f0f6-4d49-b519-9d30746433ca"). InnerVolumeSpecName "kube-api-access-5t574". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:38.131832 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.131751    6014 scope.go:122] "RemoveContainer" containerID="8a404236ae55b1b8ac99a51fd7e7315ffa5d364d5709cbac5896816ecee9cffc"
Oct 30 19:16:38.132208 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:16:38.132117    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a404236ae55b1b8ac99a51fd7e7315ffa5d364d5709cbac5896816ecee9cffc\": container with ID starting with 8a404236ae55b1b8ac99a51fd7e7315ffa5d364d5709cbac5896816ecee9cffc not found: ID does not exist" containerID="8a404236ae55b1b8ac99a51fd7e7315ffa5d364d5709cbac5896816ecee9cffc"
Oct 30 19:16:38.132208 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.132191    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a404236ae55b1b8ac99a51fd7e7315ffa5d364d5709cbac5896816ecee9cffc"} err="failed to get container status \"8a404236ae55b1b8ac99a51fd7e7315ffa5d364d5709cbac5896816ecee9cffc\": rpc error: code = NotFound desc = could not find container \"8a404236ae55b1b8ac99a51fd7e7315ffa5d364d5709cbac5896816ecee9cffc\": container with ID starting with 8a404236ae55b1b8ac99a51fd7e7315ffa5d364d5709cbac5896816ecee9cffc not found: ID does not exist"
Oct 30 19:16:38.196871 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.196841    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"
Oct 30 19:16:38.228284 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.228206    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b68589-f0f6-4d49-b519-9d30746433ca-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:38.228284 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.228242    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t574\" (UniqueName: \"kubernetes.io/projected/97b68589-f0f6-4d49-b519-9d30746433ca-kube-api-access-5t574\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:38.294112 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.294052    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm" podStartSLOduration=1.294035961 podStartE2EDuration="1.294035961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:38.291593249 +0000 UTC m=+2236.934760865" watchObservedRunningTime="2026-10-30 19:16:38.294035961 +0000 UTC m=+2236.937203575"
Oct 30 19:16:38.328899 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.328855    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5eebbf-70d4-48fe-827a-965f191a8938-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5eebbf-70d4-48fe-827a-965f191a8938-plugins-config-volume\") pod \"ea5eebbf-70d4-48fe-827a-965f191a8938\" (UID: \"ea5eebbf-70d4-48fe-827a-965f191a8938\") "
Oct 30 19:16:38.329109 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.329019    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5eebbf-70d4-48fe-827a-965f191a8938-kube-api-access-p4fbc\" (UniqueName: \"kubernetes.io/projected/ea5eebbf-70d4-48fe-827a-965f191a8938-kube-api-access-p4fbc\") pod \"ea5eebbf-70d4-48fe-827a-965f191a8938\" (UID: \"ea5eebbf-70d4-48fe-827a-965f191a8938\") "
Oct 30 19:16:38.329306 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.329267    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5eebbf-70d4-48fe-827a-965f191a8938-plugins-config-volume" pod "ea5eebbf-70d4-48fe-827a-965f191a8938" (UID: "ea5eebbf-70d4-48fe-827a-965f191a8938"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:38.329647 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.329628    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5eebbf-70d4-48fe-827a-965f191a8938-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:38.331477 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.331449    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5eebbf-70d4-48fe-827a-965f191a8938-kube-api-access-p4fbc" pod "ea5eebbf-70d4-48fe-827a-965f191a8938" (UID: "ea5eebbf-70d4-48fe-827a-965f191a8938"). InnerVolumeSpecName "kube-api-access-p4fbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:38.431264 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.431231    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4fbc\" (UniqueName: \"kubernetes.io/projected/ea5eebbf-70d4-48fe-827a-965f191a8938-kube-api-access-p4fbc\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:38.591353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.591321    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"]
Oct 30 19:16:38.597774 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:38.597744    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-l8jk9"]
Oct 30 19:16:39.133832 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:39.133803    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"
Oct 30 19:16:39.134314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:39.133803    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6" event={"ID":"ea5eebbf-70d4-48fe-827a-965f191a8938","Type":"ContainerDied","Data":"865076b3d870e458d907104367ea1f9b0c07ca71a79ca8791fbacd0f3633aabe"}
Oct 30 19:16:39.134314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:39.133959    6014 scope.go:122] "RemoveContainer" containerID="7a9be7426a674d37965faca4f3cb50e9036373362f4fe5122f6bbdb5937ad8fc"
Oct 30 19:16:39.135961 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:39.135933    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk" event={"ID":"b420a535-af91-4068-ae30-fef541da631b","Type":"ContainerStarted","Data":"e6f4e5d20b3f25bbb6949e61dac3b388408e4a4aa37031d64e8bc09afd786598"}
Oct 30 19:16:39.136201 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:39.136180    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"
Oct 30 19:16:39.302156 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:39.302091    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk" podStartSLOduration=2.302074933 podStartE2EDuration="2.302074933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:39.299910587 +0000 UTC m=+2237.943078201" watchObservedRunningTime="2026-10-30 19:16:39.302074933 +0000 UTC m=+2237.945242571"
Oct 30 19:16:39.428224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:39.424866    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"]
Oct 30 19:16:39.436584 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:39.436542    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae6a138d-5544db4579-sv8f6"]
Oct 30 19:16:39.477379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:39.477346    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b68589-f0f6-4d49-b519-9d30746433ca" path="/var/lib/kubelet/pods/97b68589-f0f6-4d49-b519-9d30746433ca/volumes"
Oct 30 19:16:39.477865 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:39.477844    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5eebbf-70d4-48fe-827a-965f191a8938" path="/var/lib/kubelet/pods/ea5eebbf-70d4-48fe-827a-965f191a8938/volumes"
Oct 30 19:16:50.149398 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:50.149367    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"
Oct 30 19:16:50.149870 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:50.149444    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"
Oct 30 19:16:53.259919 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:16:53.259877    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:16:56.864241 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:16:56.864206    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:17:01.320036 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.319952    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"]
Oct 30 19:17:01.320586 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.320196    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp" podUID="7301bcc9-644f-4488-aa20-6405d959b866" containerName="maas-api" containerID="cri-o://87c960c01c1c0dec605ff7bfb79b81bbcb021cf0bfb37320fd4844da53f2842b" gracePeriod=30
Oct 30 19:17:01.614610 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.614577    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:17:01.775481 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.775444    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7301bcc9-644f-4488-aa20-6405d959b866-kube-api-access-p2k4k\" (UniqueName: \"kubernetes.io/projected/7301bcc9-644f-4488-aa20-6405d959b866-kube-api-access-p2k4k\") pod \"7301bcc9-644f-4488-aa20-6405d959b866\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") "
Oct 30 19:17:01.775688 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.775560    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-metrics-certs\") pod \"7301bcc9-644f-4488-aa20-6405d959b866\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") "
Oct 30 19:17:01.775770 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.775720    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-maas-api-tls\") pod \"7301bcc9-644f-4488-aa20-6405d959b866\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") "
Oct 30 19:17:01.775836 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.775776    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7301bcc9-644f-4488-aa20-6405d959b866-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7301bcc9-644f-4488-aa20-6405d959b866-openshift-service-ca\") pod \"7301bcc9-644f-4488-aa20-6405d959b866\" (UID: \"7301bcc9-644f-4488-aa20-6405d959b866\") "
Oct 30 19:17:01.776937 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.776897    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7301bcc9-644f-4488-aa20-6405d959b866-openshift-service-ca" pod "7301bcc9-644f-4488-aa20-6405d959b866" (UID: "7301bcc9-644f-4488-aa20-6405d959b866"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:01.778719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.778688    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7301bcc9-644f-4488-aa20-6405d959b866-kube-api-access-p2k4k" pod "7301bcc9-644f-4488-aa20-6405d959b866" (UID: "7301bcc9-644f-4488-aa20-6405d959b866"). InnerVolumeSpecName "kube-api-access-p2k4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:01.778864 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.778741    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-maas-api-tls" pod "7301bcc9-644f-4488-aa20-6405d959b866" (UID: "7301bcc9-644f-4488-aa20-6405d959b866"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:01.779467 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.779435    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-metrics-certs" pod "7301bcc9-644f-4488-aa20-6405d959b866" (UID: "7301bcc9-644f-4488-aa20-6405d959b866"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:01.883875 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.883763    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:01.883875 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.883810    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7301bcc9-644f-4488-aa20-6405d959b866-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:01.883875 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.883826    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7301bcc9-644f-4488-aa20-6405d959b866-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:01.883875 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:01.883840    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2k4k\" (UniqueName: \"kubernetes.io/projected/7301bcc9-644f-4488-aa20-6405d959b866-kube-api-access-p2k4k\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:02.401776 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:02.401747    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"
Oct 30 19:17:02.406858 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:02.405678    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp" event={"ID":"7301bcc9-644f-4488-aa20-6405d959b866","Type":"ContainerDied","Data":"87c960c01c1c0dec605ff7bfb79b81bbcb021cf0bfb37320fd4844da53f2842b"}
Oct 30 19:17:02.406858 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:02.405756    6014 scope.go:122] "RemoveContainer" containerID="87c960c01c1c0dec605ff7bfb79b81bbcb021cf0bfb37320fd4844da53f2842b"
Oct 30 19:17:02.407629 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:02.407216    6014 generic.go:356] "Generic (PLEG): container finished" podID="7301bcc9-644f-4488-aa20-6405d959b866" containerID="87c960c01c1c0dec605ff7bfb79b81bbcb021cf0bfb37320fd4844da53f2842b" exitCode=0
Oct 30 19:17:02.407629 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:02.407346    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp" event={"ID":"7301bcc9-644f-4488-aa20-6405d959b866","Type":"ContainerDied","Data":"1c993da102ead301971910505d90d76e02a69bc61a52c667ac2d34b4784af2c4"}
Oct 30 19:17:02.420356 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:02.420331    6014 scope.go:122] "RemoveContainer" containerID="87c960c01c1c0dec605ff7bfb79b81bbcb021cf0bfb37320fd4844da53f2842b"
Oct 30 19:17:02.420696 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:17:02.420675    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c960c01c1c0dec605ff7bfb79b81bbcb021cf0bfb37320fd4844da53f2842b\": container with ID starting with 87c960c01c1c0dec605ff7bfb79b81bbcb021cf0bfb37320fd4844da53f2842b not found: ID does not exist" containerID="87c960c01c1c0dec605ff7bfb79b81bbcb021cf0bfb37320fd4844da53f2842b"
Oct 30 19:17:02.420754 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:02.420706    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c960c01c1c0dec605ff7bfb79b81bbcb021cf0bfb37320fd4844da53f2842b"} err="failed to get container status \"87c960c01c1c0dec605ff7bfb79b81bbcb021cf0bfb37320fd4844da53f2842b\": rpc error: code = NotFound desc = could not find container \"87c960c01c1c0dec605ff7bfb79b81bbcb021cf0bfb37320fd4844da53f2842b\": container with ID starting with 87c960c01c1c0dec605ff7bfb79b81bbcb021cf0bfb37320fd4844da53f2842b not found: ID does not exist"
Oct 30 19:17:02.569587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:02.569551    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"]
Oct 30 19:17:02.574498 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:02.574472    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8da6e2-5b95fd7677-jdngp"]
Oct 30 19:17:03.477091 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:03.477055    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7301bcc9-644f-4488-aa20-6405d959b866" path="/var/lib/kubelet/pods/7301bcc9-644f-4488-aa20-6405d959b866/volumes"
Oct 30 19:17:07.502628 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:07.502595    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"]
Oct 30 19:17:07.503130 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:07.502793    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq" podUID="f52ebf7a-2923-4b71-9989-0f9029fc6dfc" containerName="maas-api" containerID="cri-o://7c998dbedde824ee2accb29dbdbcc0ef257546795e9bc425f50c19d1c838bf4c" gracePeriod=30
Oct 30 19:17:07.819429 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:07.819400    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:17:07.916165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:07.898813    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-kube-api-access-lthfv\" (UniqueName: \"kubernetes.io/projected/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-kube-api-access-lthfv\") pod \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") "
Oct 30 19:17:07.916165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:07.899126    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-metrics-certs\") pod \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") "
Oct 30 19:17:07.916165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:07.899250    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-openshift-service-ca\") pod \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") "
Oct 30 19:17:07.916165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:07.899308    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-maas-api-tls\") pod \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\" (UID: \"f52ebf7a-2923-4b71-9989-0f9029fc6dfc\") "
Oct 30 19:17:07.956162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:07.946357    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-openshift-service-ca" pod "f52ebf7a-2923-4b71-9989-0f9029fc6dfc" (UID: "f52ebf7a-2923-4b71-9989-0f9029fc6dfc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:07.976418 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:07.964330    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-metrics-certs" pod "f52ebf7a-2923-4b71-9989-0f9029fc6dfc" (UID: "f52ebf7a-2923-4b71-9989-0f9029fc6dfc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:08.022164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.009836    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-kube-api-access-lthfv" pod "f52ebf7a-2923-4b71-9989-0f9029fc6dfc" (UID: "f52ebf7a-2923-4b71-9989-0f9029fc6dfc"). InnerVolumeSpecName "kube-api-access-lthfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:08.022164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.010206    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-maas-api-tls" pod "f52ebf7a-2923-4b71-9989-0f9029fc6dfc" (UID: "f52ebf7a-2923-4b71-9989-0f9029fc6dfc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:08.030210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.023861    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:08.030594 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.030516    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:08.030594 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.030551    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lthfv\" (UniqueName: \"kubernetes.io/projected/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-kube-api-access-lthfv\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:08.030594 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.030566    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52ebf7a-2923-4b71-9989-0f9029fc6dfc-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:08.495162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.486851    6014 generic.go:356] "Generic (PLEG): container finished" podID="f52ebf7a-2923-4b71-9989-0f9029fc6dfc" containerID="7c998dbedde824ee2accb29dbdbcc0ef257546795e9bc425f50c19d1c838bf4c" exitCode=0
Oct 30 19:17:08.495162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.486999    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq" event={"ID":"f52ebf7a-2923-4b71-9989-0f9029fc6dfc","Type":"ContainerDied","Data":"7c998dbedde824ee2accb29dbdbcc0ef257546795e9bc425f50c19d1c838bf4c"}
Oct 30 19:17:08.495162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.487057    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq" event={"ID":"f52ebf7a-2923-4b71-9989-0f9029fc6dfc","Type":"ContainerDied","Data":"256e9e5d844dfb09b2fcdddf4035b1a3b93d4b51f3d19de5a0952effa4d67b54"}
Oct 30 19:17:08.495162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.487091    6014 scope.go:122] "RemoveContainer" containerID="7c998dbedde824ee2accb29dbdbcc0ef257546795e9bc425f50c19d1c838bf4c"
Oct 30 19:17:08.495162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.487273    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"
Oct 30 19:17:08.566394 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.566371    6014 scope.go:122] "RemoveContainer" containerID="7c998dbedde824ee2accb29dbdbcc0ef257546795e9bc425f50c19d1c838bf4c"
Oct 30 19:17:08.566816 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:17:08.566788    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c998dbedde824ee2accb29dbdbcc0ef257546795e9bc425f50c19d1c838bf4c\": container with ID starting with 7c998dbedde824ee2accb29dbdbcc0ef257546795e9bc425f50c19d1c838bf4c not found: ID does not exist" containerID="7c998dbedde824ee2accb29dbdbcc0ef257546795e9bc425f50c19d1c838bf4c"
Oct 30 19:17:08.566892 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.566829    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c998dbedde824ee2accb29dbdbcc0ef257546795e9bc425f50c19d1c838bf4c"} err="failed to get container status \"7c998dbedde824ee2accb29dbdbcc0ef257546795e9bc425f50c19d1c838bf4c\": rpc error: code = NotFound desc = could not find container \"7c998dbedde824ee2accb29dbdbcc0ef257546795e9bc425f50c19d1c838bf4c\": container with ID starting with 7c998dbedde824ee2accb29dbdbcc0ef257546795e9bc425f50c19d1c838bf4c not found: ID does not exist"
Oct 30 19:17:08.678214 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.678179    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"]
Oct 30 19:17:08.725755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:08.725721    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f26b1d-677c775cd5-xwmhq"]
Oct 30 19:17:09.477267 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:09.477238    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f52ebf7a-2923-4b71-9989-0f9029fc6dfc" path="/var/lib/kubelet/pods/f52ebf7a-2923-4b71-9989-0f9029fc6dfc/volumes"
Oct 30 19:17:16.862296 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:16.862256    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"]
Oct 30 19:17:16.862783 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:16.862564    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerName="istio-proxy" containerID="cri-o://a6a88c8725b79fa1ad933e1ea48477321d71bdfe3b5a7ade02de5f44e3adf157" gracePeriod=30
Oct 30 19:17:17.623278 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:17.623232    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"]
Oct 30 19:17:20.344793 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.344758    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"]
Oct 30 19:17:20.345237 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.344981    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg" podUID="455c906c-c097-4889-ac72-4c1dbc6b82d8" containerName="maas-api" containerID="cri-o://b83d41743f3495f4fae6f20c58bfd929194249fd41fbbc93537ab039cf82e9b3" gracePeriod=30
Oct 30 19:17:20.597695 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.597594    6014 generic.go:356] "Generic (PLEG): container finished" podID="455c906c-c097-4889-ac72-4c1dbc6b82d8" containerID="b83d41743f3495f4fae6f20c58bfd929194249fd41fbbc93537ab039cf82e9b3" exitCode=0
Oct 30 19:17:20.597874 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.597761    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg" event={"ID":"455c906c-c097-4889-ac72-4c1dbc6b82d8","Type":"ContainerDied","Data":"b83d41743f3495f4fae6f20c58bfd929194249fd41fbbc93537ab039cf82e9b3"}
Oct 30 19:17:20.649637 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.649608    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:17:20.793187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.788701    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/455c906c-c097-4889-ac72-4c1dbc6b82d8-kube-api-access-zq995\" (UniqueName: \"kubernetes.io/projected/455c906c-c097-4889-ac72-4c1dbc6b82d8-kube-api-access-zq995\") pod \"455c906c-c097-4889-ac72-4c1dbc6b82d8\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") "
Oct 30 19:17:20.793187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.788819    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-metrics-certs\") pod \"455c906c-c097-4889-ac72-4c1dbc6b82d8\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") "
Oct 30 19:17:20.793187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.788868    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-maas-api-tls\") pod \"455c906c-c097-4889-ac72-4c1dbc6b82d8\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") "
Oct 30 19:17:20.793187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.789002    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/455c906c-c097-4889-ac72-4c1dbc6b82d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/455c906c-c097-4889-ac72-4c1dbc6b82d8-openshift-service-ca\") pod \"455c906c-c097-4889-ac72-4c1dbc6b82d8\" (UID: \"455c906c-c097-4889-ac72-4c1dbc6b82d8\") "
Oct 30 19:17:20.793187 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.790462    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455c906c-c097-4889-ac72-4c1dbc6b82d8-openshift-service-ca" pod "455c906c-c097-4889-ac72-4c1dbc6b82d8" (UID: "455c906c-c097-4889-ac72-4c1dbc6b82d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:20.809163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.796676    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455c906c-c097-4889-ac72-4c1dbc6b82d8-kube-api-access-zq995" pod "455c906c-c097-4889-ac72-4c1dbc6b82d8" (UID: "455c906c-c097-4889-ac72-4c1dbc6b82d8"). InnerVolumeSpecName "kube-api-access-zq995". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:20.831821 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.831753    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-maas-api-tls" pod "455c906c-c097-4889-ac72-4c1dbc6b82d8" (UID: "455c906c-c097-4889-ac72-4c1dbc6b82d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:20.833005 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.832975    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-metrics-certs" pod "455c906c-c097-4889-ac72-4c1dbc6b82d8" (UID: "455c906c-c097-4889-ac72-4c1dbc6b82d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:20.891631 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.891536    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/455c906c-c097-4889-ac72-4c1dbc6b82d8-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:20.891631 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.891572    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq995\" (UniqueName: \"kubernetes.io/projected/455c906c-c097-4889-ac72-4c1dbc6b82d8-kube-api-access-zq995\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:20.891631 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.891588    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:20.891631 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:20.891602    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/455c906c-c097-4889-ac72-4c1dbc6b82d8-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:21.609580 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:21.609541    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg" event={"ID":"455c906c-c097-4889-ac72-4c1dbc6b82d8","Type":"ContainerDied","Data":"94de94a195b259a5dd9848edb84b6ae4ef42d85d6eab2bd38e85194f2234b99e"}
Oct 30 19:17:21.610059 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:21.609619    6014 scope.go:122] "RemoveContainer" containerID="b83d41743f3495f4fae6f20c58bfd929194249fd41fbbc93537ab039cf82e9b3"
Oct 30 19:17:21.610059 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:21.609677    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"
Oct 30 19:17:21.773403 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:21.773312    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"]
Oct 30 19:17:21.822610 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:21.822576    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-87ed12-78f7d458-ffwrg"]
Oct 30 19:17:22.158084 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.158054    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:17:22.214949 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.214914    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-credential-socket\") pod \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") "
Oct 30 19:17:22.215184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.214964    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-envoy\") pod \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") "
Oct 30 19:17:22.215184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.215070    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-kube-api-access-z7pcd\" (UniqueName: \"kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-kube-api-access-z7pcd\") pod \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") "
Oct 30 19:17:22.215184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.215158    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-certs\") pod \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") "
Oct 30 19:17:22.215392 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.215186    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istiod-ca-cert\") pod \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") "
Oct 30 19:17:22.215392 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.215214    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-podinfo\") pod \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") "
Oct 30 19:17:22.215392 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.215264    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-socket\") pod \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") "
Oct 30 19:17:22.215392 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.215268    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-credential-socket" pod "69d8cf2e-7a19-49f8-8967-2deeff9ed05c" (UID: "69d8cf2e-7a19-49f8-8967-2deeff9ed05c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:22.215392 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.215298    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-data\") pod \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") "
Oct 30 19:17:22.215648 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.215502    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-token\" (UniqueName: \"kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-token\") pod \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\" (UID: \"69d8cf2e-7a19-49f8-8967-2deeff9ed05c\") "
Oct 30 19:17:22.215648 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.215505    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-certs" pod "69d8cf2e-7a19-49f8-8967-2deeff9ed05c" (UID: "69d8cf2e-7a19-49f8-8967-2deeff9ed05c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:22.215648 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.215614    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-socket" pod "69d8cf2e-7a19-49f8-8967-2deeff9ed05c" (UID: "69d8cf2e-7a19-49f8-8967-2deeff9ed05c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:22.215648 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.215633    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istiod-ca-cert" pod "69d8cf2e-7a19-49f8-8967-2deeff9ed05c" (UID: "69d8cf2e-7a19-49f8-8967-2deeff9ed05c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:22.215829 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.215751    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-data" pod "69d8cf2e-7a19-49f8-8967-2deeff9ed05c" (UID: "69d8cf2e-7a19-49f8-8967-2deeff9ed05c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:22.216443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.216419    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:22.216568 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.216447    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:22.216568 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.216462    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:22.216568 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.216476    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:22.216568 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.216494    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:22.217440 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.217419    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-envoy" pod "69d8cf2e-7a19-49f8-8967-2deeff9ed05c" (UID: "69d8cf2e-7a19-49f8-8967-2deeff9ed05c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:22.217657 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.217637    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-podinfo" pod "69d8cf2e-7a19-49f8-8967-2deeff9ed05c" (UID: "69d8cf2e-7a19-49f8-8967-2deeff9ed05c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:22.217744 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.217665    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-kube-api-access-z7pcd" pod "69d8cf2e-7a19-49f8-8967-2deeff9ed05c" (UID: "69d8cf2e-7a19-49f8-8967-2deeff9ed05c"). InnerVolumeSpecName "kube-api-access-z7pcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:22.217937 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.217922    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-token" pod "69d8cf2e-7a19-49f8-8967-2deeff9ed05c" (UID: "69d8cf2e-7a19-49f8-8967-2deeff9ed05c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:22.318058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.318018    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7pcd\" (UniqueName: \"kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-kube-api-access-z7pcd\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:22.318058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.318054    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:22.318297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.318069    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:22.318297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.318084    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d8cf2e-7a19-49f8-8967-2deeff9ed05c-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:22.619904 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.619814    6014 generic.go:356] "Generic (PLEG): container finished" podID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerID="a6a88c8725b79fa1ad933e1ea48477321d71bdfe3b5a7ade02de5f44e3adf157" exitCode=0
Oct 30 19:17:22.619904 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.619858    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" event={"ID":"69d8cf2e-7a19-49f8-8967-2deeff9ed05c","Type":"ContainerDied","Data":"a6a88c8725b79fa1ad933e1ea48477321d71bdfe3b5a7ade02de5f44e3adf157"}
Oct 30 19:17:22.619904 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.619884    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"
Oct 30 19:17:22.619904 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.619900    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc" event={"ID":"69d8cf2e-7a19-49f8-8967-2deeff9ed05c","Type":"ContainerDied","Data":"cb2222da3d91a5e3e9f2fad6245cb4d13b87f79afadcf76f6b0eee0daa3608a3"}
Oct 30 19:17:22.620609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.619927    6014 scope.go:122] "RemoveContainer" containerID="a6a88c8725b79fa1ad933e1ea48477321d71bdfe3b5a7ade02de5f44e3adf157"
Oct 30 19:17:22.633431 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.633412    6014 scope.go:122] "RemoveContainer" containerID="61437ab78b927ffa560c1d02fc53c5f1fd9f3b0e6a4e804208b27d7b89eb9c39"
Oct 30 19:17:22.644130 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.644056    6014 scope.go:122] "RemoveContainer" containerID="a6a88c8725b79fa1ad933e1ea48477321d71bdfe3b5a7ade02de5f44e3adf157"
Oct 30 19:17:22.644505 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:17:22.644469    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a88c8725b79fa1ad933e1ea48477321d71bdfe3b5a7ade02de5f44e3adf157\": container with ID starting with a6a88c8725b79fa1ad933e1ea48477321d71bdfe3b5a7ade02de5f44e3adf157 not found: ID does not exist" containerID="a6a88c8725b79fa1ad933e1ea48477321d71bdfe3b5a7ade02de5f44e3adf157"
Oct 30 19:17:22.644641 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.644516    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a88c8725b79fa1ad933e1ea48477321d71bdfe3b5a7ade02de5f44e3adf157"} err="failed to get container status \"a6a88c8725b79fa1ad933e1ea48477321d71bdfe3b5a7ade02de5f44e3adf157\": rpc error: code = NotFound desc = could not find container \"a6a88c8725b79fa1ad933e1ea48477321d71bdfe3b5a7ade02de5f44e3adf157\": container with ID starting with a6a88c8725b79fa1ad933e1ea48477321d71bdfe3b5a7ade02de5f44e3adf157 not found: ID does not exist"
Oct 30 19:17:22.644641 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.644544    6014 scope.go:122] "RemoveContainer" containerID="61437ab78b927ffa560c1d02fc53c5f1fd9f3b0e6a4e804208b27d7b89eb9c39"
Oct 30 19:17:22.646348 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:17:22.646325    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61437ab78b927ffa560c1d02fc53c5f1fd9f3b0e6a4e804208b27d7b89eb9c39\": container with ID starting with 61437ab78b927ffa560c1d02fc53c5f1fd9f3b0e6a4e804208b27d7b89eb9c39 not found: ID does not exist" containerID="61437ab78b927ffa560c1d02fc53c5f1fd9f3b0e6a4e804208b27d7b89eb9c39"
Oct 30 19:17:22.646452 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.646354    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61437ab78b927ffa560c1d02fc53c5f1fd9f3b0e6a4e804208b27d7b89eb9c39"} err="failed to get container status \"61437ab78b927ffa560c1d02fc53c5f1fd9f3b0e6a4e804208b27d7b89eb9c39\": rpc error: code = NotFound desc = could not find container \"61437ab78b927ffa560c1d02fc53c5f1fd9f3b0e6a4e804208b27d7b89eb9c39\": container with ID starting with 61437ab78b927ffa560c1d02fc53c5f1fd9f3b0e6a4e804208b27d7b89eb9c39 not found: ID does not exist"
Oct 30 19:17:22.826270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.826196    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"]
Oct 30 19:17:22.861177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:22.858381    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f26b1d-openshift-default-f656f8447-z77nc"]
Oct 30 19:17:23.259905 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:17:23.259871    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:17:23.493576 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:23.493544    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="455c906c-c097-4889-ac72-4c1dbc6b82d8" path="/var/lib/kubelet/pods/455c906c-c097-4889-ac72-4c1dbc6b82d8/volumes"
Oct 30 19:17:23.494165 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:23.494127    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" path="/var/lib/kubelet/pods/69d8cf2e-7a19-49f8-8967-2deeff9ed05c/volumes"
Oct 30 19:17:24.455569 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:24.455530    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"]
Oct 30 19:17:29.450596 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:29.450557    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"]
Oct 30 19:17:29.451050 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:29.450829    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5" podUID="111205d6-93ca-400e-8383-1b1c555d47dc" containerName="istio-proxy" containerID="cri-o://434ae9326b9232252ee465690ad10e392ca7c3488607dd8e88d0b4c7cb1d7c8e" gracePeriod=30
Oct 30 19:17:29.456747 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:29.456712    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb" podUID="cbd18ae9-9e76-42e1-bde3-da20fcc21ad8" containerName="payload-pre-processing" containerID="cri-o://674d30df0aa1a1d439ddee5f2bd20898e0bff436d87d288357e7093515a1f121" gracePeriod=25
Oct 30 19:17:30.249197 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.249161    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"]
Oct 30 19:17:30.249921 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.249900    6014 cpu_manager.go:414] "containerMap: removing container" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerName="istio-proxy"
Oct 30 19:17:30.250058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.249923    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerName="istio-proxy"
Oct 30 19:17:30.250058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.249965    6014 cpu_manager.go:414] "containerMap: removing container" podUID="f52ebf7a-2923-4b71-9989-0f9029fc6dfc" containerName="maas-api"
Oct 30 19:17:30.250058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.249973    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52ebf7a-2923-4b71-9989-0f9029fc6dfc" containerName="maas-api"
Oct 30 19:17:30.250058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.249982    6014 cpu_manager.go:414] "containerMap: removing container" podUID="97b68589-f0f6-4d49-b519-9d30746433ca" containerName="payload-processing"
Oct 30 19:17:30.250058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.249989    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b68589-f0f6-4d49-b519-9d30746433ca" containerName="payload-processing"
Oct 30 19:17:30.250058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.250010    6014 cpu_manager.go:414] "containerMap: removing container" podUID="455c906c-c097-4889-ac72-4c1dbc6b82d8" containerName="maas-api"
Oct 30 19:17:30.250058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.250017    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455c906c-c097-4889-ac72-4c1dbc6b82d8" containerName="maas-api"
Oct 30 19:17:30.250058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.250060    6014 cpu_manager.go:414] "containerMap: removing container" podUID="7301bcc9-644f-4488-aa20-6405d959b866" containerName="maas-api"
Oct 30 19:17:30.250515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.250067    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7301bcc9-644f-4488-aa20-6405d959b866" containerName="maas-api"
Oct 30 19:17:30.250515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.250100    6014 cpu_manager.go:414] "containerMap: removing container" podUID="ea5eebbf-70d4-48fe-827a-965f191a8938" containerName="payload-pre-processing"
Oct 30 19:17:30.250515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.250108    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5eebbf-70d4-48fe-827a-965f191a8938" containerName="payload-pre-processing"
Oct 30 19:17:30.250515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.250307    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerName="istio-proxy"
Oct 30 19:17:30.250515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.250329    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5eebbf-70d4-48fe-827a-965f191a8938" containerName="payload-pre-processing"
Oct 30 19:17:30.250515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.250340    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerName="istio-proxy"
Oct 30 19:17:30.250515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.250356    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52ebf7a-2923-4b71-9989-0f9029fc6dfc" containerName="maas-api"
Oct 30 19:17:30.250515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.250365    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b68589-f0f6-4d49-b519-9d30746433ca" containerName="payload-processing"
Oct 30 19:17:30.250515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.250376    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="455c906c-c097-4889-ac72-4c1dbc6b82d8" containerName="maas-api"
Oct 30 19:17:30.250515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.250392    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7301bcc9-644f-4488-aa20-6405d959b866" containerName="maas-api"
Oct 30 19:17:30.256220 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.256197    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"
Oct 30 19:17:30.292132 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.292095    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6g4z\" (UniqueName: \"kubernetes.io/projected/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-kube-api-access-w6g4z\") pod \"maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g\" (UID: \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"
Oct 30 19:17:30.292304 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.292246    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g\" (UID: \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"
Oct 30 19:17:30.292361 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.292328    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g\" (UID: \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"
Oct 30 19:17:30.309964 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.309928    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"]
Oct 30 19:17:30.355407 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.355365    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:17:30.355407 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.355366    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:30.393325 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.393291    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g\" (UID: \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"
Oct 30 19:17:30.393527 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.393510    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6g4z\" (UniqueName: \"kubernetes.io/projected/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-kube-api-access-w6g4z\") pod \"maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g\" (UID: \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"
Oct 30 19:17:30.393591 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.393575    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g\" (UID: \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"
Oct 30 19:17:30.394262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.394237    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g\" (UID: \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"
Oct 30 19:17:30.395814 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.395792    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g\" (UID: \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"
Oct 30 19:17:30.491162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.476261    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6g4z\" (UniqueName: \"kubernetes.io/projected/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-kube-api-access-w6g4z\") pod \"maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g\" (UID: \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"
Oct 30 19:17:30.570053 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.570017    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"
Oct 30 19:17:30.957071 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:30.957037    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"]
Oct 30 19:17:30.958899 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:17:30.958872    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152f91b3_d4d9_4cf2_a821_12062ad0fb9b.slice/crio-fc8ad3995529ef57a0a59312c48bddd499cea8db0e26993041d6814babb910b6 WatchSource:0}: Error finding container fc8ad3995529ef57a0a59312c48bddd499cea8db0e26993041d6814babb910b6: Status 404 returned error can't find the container with id fc8ad3995529ef57a0a59312c48bddd499cea8db0e26993041d6814babb910b6
Oct 30 19:17:31.720635 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:31.720594    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g" event={"ID":"152f91b3-d4d9-4cf2-a821-12062ad0fb9b","Type":"ContainerStarted","Data":"5d3a3b2089aac09593448a67d3fc09c13fa146beca35d0d12b2ab594ebfc6822"}
Oct 30 19:17:31.721050 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:31.720650    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g" event={"ID":"152f91b3-d4d9-4cf2-a821-12062ad0fb9b","Type":"ContainerStarted","Data":"fc8ad3995529ef57a0a59312c48bddd499cea8db0e26993041d6814babb910b6"}
Oct 30 19:17:31.837955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:31.837905    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g" podStartSLOduration=1.837891223 podStartE2EDuration="1.837891223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:31.834929933 +0000 UTC m=+2290.478097546" watchObservedRunningTime="2026-10-30 19:17:31.837891223 +0000 UTC m=+2290.481058863"
Oct 30 19:17:32.651228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.651186    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main" containerID="cri-o://e3ff3c2e842a68a60eeff2c2e04e0a71298f949ae3aa587dc2730fd64e49d5a8" gracePeriod=45
Oct 30 19:17:32.728474 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.728436    6014 generic.go:356] "Generic (PLEG): container finished" podID="152f91b3-d4d9-4cf2-a821-12062ad0fb9b" containerID="5d3a3b2089aac09593448a67d3fc09c13fa146beca35d0d12b2ab594ebfc6822" exitCode=0
Oct 30 19:17:32.728934 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.728518    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g" event={"ID":"152f91b3-d4d9-4cf2-a821-12062ad0fb9b","Type":"ContainerDied","Data":"5d3a3b2089aac09593448a67d3fc09c13fa146beca35d0d12b2ab594ebfc6822"}
Oct 30 19:17:32.935807 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.935779    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:17:32.946510 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.946483    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/799cb959-41de-49dd-bab4-d64a2d054208-tls-certs\" (UniqueName: \"kubernetes.io/secret/799cb959-41de-49dd-bab4-d64a2d054208-tls-certs\") pod \"799cb959-41de-49dd-bab4-d64a2d054208\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") "
Oct 30 19:17:32.946635 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.946582    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-model-cache\") pod \"799cb959-41de-49dd-bab4-d64a2d054208\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") "
Oct 30 19:17:32.946635 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.946625    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799cb959-41de-49dd-bab4-d64a2d054208-kube-api-access-jsvgk\" (UniqueName: \"kubernetes.io/projected/799cb959-41de-49dd-bab4-d64a2d054208-kube-api-access-jsvgk\") pod \"799cb959-41de-49dd-bab4-d64a2d054208\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") "
Oct 30 19:17:32.946712 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.946659    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-tmp-dir\") pod \"799cb959-41de-49dd-bab4-d64a2d054208\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") "
Oct 30 19:17:32.946831 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.946814    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-dshm\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-dshm\") pod \"799cb959-41de-49dd-bab4-d64a2d054208\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") "
Oct 30 19:17:32.946887 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.946856    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-model-cache" pod "799cb959-41de-49dd-bab4-d64a2d054208" (UID: "799cb959-41de-49dd-bab4-d64a2d054208"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:32.946976 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.946955    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-home\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-home\") pod \"799cb959-41de-49dd-bab4-d64a2d054208\" (UID: \"799cb959-41de-49dd-bab4-d64a2d054208\") "
Oct 30 19:17:32.947043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.946969    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-tmp-dir" pod "799cb959-41de-49dd-bab4-d64a2d054208" (UID: "799cb959-41de-49dd-bab4-d64a2d054208"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:32.947330 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.947233    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-home" pod "799cb959-41de-49dd-bab4-d64a2d054208" (UID: "799cb959-41de-49dd-bab4-d64a2d054208"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:32.947780 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.947759    6014 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-home\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:32.947869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.947786    6014 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-model-cache\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:32.947869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.947804    6014 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-tmp-dir\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:32.949025 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.948997    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-dshm" pod "799cb959-41de-49dd-bab4-d64a2d054208" (UID: "799cb959-41de-49dd-bab4-d64a2d054208"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:32.949116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.949026    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/799cb959-41de-49dd-bab4-d64a2d054208-tls-certs" pod "799cb959-41de-49dd-bab4-d64a2d054208" (UID: "799cb959-41de-49dd-bab4-d64a2d054208"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:32.949116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:32.949074    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799cb959-41de-49dd-bab4-d64a2d054208-kube-api-access-jsvgk" pod "799cb959-41de-49dd-bab4-d64a2d054208" (UID: "799cb959-41de-49dd-bab4-d64a2d054208"). InnerVolumeSpecName "kube-api-access-jsvgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:33.080587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.065185    6014 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/799cb959-41de-49dd-bab4-d64a2d054208-dshm\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.080587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.065224    6014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799cb959-41de-49dd-bab4-d64a2d054208-tls-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.080587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.065239    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsvgk\" (UniqueName: \"kubernetes.io/projected/799cb959-41de-49dd-bab4-d64a2d054208-kube-api-access-jsvgk\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.615404 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.615362    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"]
Oct 30 19:17:33.616564 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.616536    6014 cpu_manager.go:414] "containerMap: removing container" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main"
Oct 30 19:17:33.616564 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.616561    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main"
Oct 30 19:17:33.616730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.616579    6014 cpu_manager.go:414] "containerMap: removing container" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerName="istio-proxy"
Oct 30 19:17:33.616730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.616585    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d8cf2e-7a19-49f8-8967-2deeff9ed05c" containerName="istio-proxy"
Oct 30 19:17:33.616730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.616722    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="799cb959-41de-49dd-bab4-d64a2d054208" containerName="main"
Oct 30 19:17:33.620887 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.620866    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:33.723242 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.723200    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-714b2c-serving-cert\""
Oct 30 19:17:33.726590 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.726557    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"]
Oct 30 19:17:33.736907 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.736836    6014 generic.go:356] "Generic (PLEG): container finished" podID="799cb959-41de-49dd-bab4-d64a2d054208" containerID="e3ff3c2e842a68a60eeff2c2e04e0a71298f949ae3aa587dc2730fd64e49d5a8" exitCode=0
Oct 30 19:17:33.736907 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.736880    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"
Oct 30 19:17:33.737430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.736925    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" event={"ID":"799cb959-41de-49dd-bab4-d64a2d054208","Type":"ContainerDied","Data":"e3ff3c2e842a68a60eeff2c2e04e0a71298f949ae3aa587dc2730fd64e49d5a8"}
Oct 30 19:17:33.737430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.736976    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r" event={"ID":"799cb959-41de-49dd-bab4-d64a2d054208","Type":"ContainerDied","Data":"a59289b25e1a44eb83aa83c8a8eea6bb8c8eda165eb7b4e986642d7963780ea3"}
Oct 30 19:17:33.737430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.737002    6014 scope.go:122] "RemoveContainer" containerID="e3ff3c2e842a68a60eeff2c2e04e0a71298f949ae3aa587dc2730fd64e49d5a8"
Oct 30 19:17:33.747543 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.747520    6014 scope.go:122] "RemoveContainer" containerID="e3ff3c2e842a68a60eeff2c2e04e0a71298f949ae3aa587dc2730fd64e49d5a8"
Oct 30 19:17:33.747913 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:17:33.747885    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ff3c2e842a68a60eeff2c2e04e0a71298f949ae3aa587dc2730fd64e49d5a8\": container with ID starting with e3ff3c2e842a68a60eeff2c2e04e0a71298f949ae3aa587dc2730fd64e49d5a8 not found: ID does not exist" containerID="e3ff3c2e842a68a60eeff2c2e04e0a71298f949ae3aa587dc2730fd64e49d5a8"
Oct 30 19:17:33.748020 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.747924    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ff3c2e842a68a60eeff2c2e04e0a71298f949ae3aa587dc2730fd64e49d5a8"} err="failed to get container status \"e3ff3c2e842a68a60eeff2c2e04e0a71298f949ae3aa587dc2730fd64e49d5a8\": rpc error: code = NotFound desc = could not find container \"e3ff3c2e842a68a60eeff2c2e04e0a71298f949ae3aa587dc2730fd64e49d5a8\": container with ID starting with e3ff3c2e842a68a60eeff2c2e04e0a71298f949ae3aa587dc2730fd64e49d5a8 not found: ID does not exist"
Oct 30 19:17:33.775436 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.774800    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-openshift-service-ca\") pod \"maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:33.775436 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.774881    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-metrics-certs\") pod \"maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:33.775436 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.775004    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9zlz\" (UniqueName: \"kubernetes.io/projected/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-kube-api-access-b9zlz\") pod \"maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:33.775436 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.775166    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-maas-api-tls\") pod \"maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:33.882302 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.881683    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-openshift-service-ca\") pod \"maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:33.882302 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.881767    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-metrics-certs\") pod \"maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:33.882302 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.881880    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9zlz\" (UniqueName: \"kubernetes.io/projected/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-kube-api-access-b9zlz\") pod \"maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:33.882302 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.881920    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-maas-api-tls\") pod \"maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:33.882302 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:17:33.882243    6014 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-serving-cert: secret "maas-api-e2e-inf-b-714b2c-serving-cert" not found
Oct 30 19:17:33.882759 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:17:33.882320    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-maas-api-tls podName:6d1c8c76-4e8d-4808-a8f4-09bd5e49f831 nodeName:}" failed. No retries permitted until 2026-10-30 19:17:34.382296656 +0000 UTC m=+2293.025464248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-maas-api-tls") pod "maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx" (UID: "6d1c8c76-4e8d-4808-a8f4-09bd5e49f831") : secret "maas-api-e2e-inf-b-714b2c-serving-cert" not found
Oct 30 19:17:33.896207 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.889602    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-openshift-service-ca\") pod \"maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:33.896207 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.893781    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-metrics-certs\") pod \"maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:33.916688 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.916650    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"]
Oct 30 19:17:33.919439 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.919419    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/distinct-2-w6-41db7c-kserve-84f956764c-wmb2r"]
Oct 30 19:17:33.949470 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.949442    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"
Oct 30 19:17:33.957239 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:33.957213    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9zlz\" (UniqueName: \"kubernetes.io/projected/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-kube-api-access-b9zlz\") pod \"maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:34.084481 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.084452    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-kube-api-access-w6g4z\" (UniqueName: \"kubernetes.io/projected/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-kube-api-access-w6g4z\") pod \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\" (UID: \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\") "
Oct 30 19:17:34.084693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.084506    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-xks-ca\") pod \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\" (UID: \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\") "
Oct 30 19:17:34.084693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.084543    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-openshift-ca\") pod \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\" (UID: \"152f91b3-d4d9-4cf2-a821-12062ad0fb9b\") "
Oct 30 19:17:34.084991 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.084966    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-openshift-ca" pod "152f91b3-d4d9-4cf2-a821-12062ad0fb9b" (UID: "152f91b3-d4d9-4cf2-a821-12062ad0fb9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:34.085839 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.085119    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.086961 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.086942    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-kube-api-access-w6g4z" pod "152f91b3-d4d9-4cf2-a821-12062ad0fb9b" (UID: "152f91b3-d4d9-4cf2-a821-12062ad0fb9b"). InnerVolumeSpecName "kube-api-access-w6g4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:34.087213 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.087185    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-xks-ca" pod "152f91b3-d4d9-4cf2-a821-12062ad0fb9b" (UID: "152f91b3-d4d9-4cf2-a821-12062ad0fb9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:34.187176 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.187101    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6g4z\" (UniqueName: \"kubernetes.io/projected/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-kube-api-access-w6g4z\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.187176 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.187174    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/152f91b3-d4d9-4cf2-a821-12062ad0fb9b-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.390108 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.390021    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-maas-api-tls\") pod \"maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:34.392633 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.392606    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-maas-api-tls\") pod \"maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:34.537132 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.537101    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:34.748248 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.748218    6014 generic.go:356] "Generic (PLEG): container finished" podID="111205d6-93ca-400e-8383-1b1c555d47dc" containerID="434ae9326b9232252ee465690ad10e392ca7c3488607dd8e88d0b4c7cb1d7c8e" exitCode=0
Oct 30 19:17:34.748674 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.748291    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5" event={"ID":"111205d6-93ca-400e-8383-1b1c555d47dc","Type":"ContainerDied","Data":"434ae9326b9232252ee465690ad10e392ca7c3488607dd8e88d0b4c7cb1d7c8e"}
Oct 30 19:17:34.750890 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.750864    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g" event={"ID":"152f91b3-d4d9-4cf2-a821-12062ad0fb9b","Type":"ContainerDied","Data":"fc8ad3995529ef57a0a59312c48bddd499cea8db0e26993041d6814babb910b6"}
Oct 30 19:17:34.751035 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.750896    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"
Oct 30 19:17:34.751035 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.750906    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc8ad3995529ef57a0a59312c48bddd499cea8db0e26993041d6814babb910b6"
Oct 30 19:17:34.768955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.768931    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:17:34.899075 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.899042    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-kube-api-access-9sr9r\" (UniqueName: \"kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-kube-api-access-9sr9r\") pod \"111205d6-93ca-400e-8383-1b1c555d47dc\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") "
Oct 30 19:17:34.899293 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.899118    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-istio-token\" (UniqueName: \"kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-istio-token\") pod \"111205d6-93ca-400e-8383-1b1c555d47dc\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") "
Oct 30 19:17:34.899293 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.899179    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-socket\") pod \"111205d6-93ca-400e-8383-1b1c555d47dc\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") "
Oct 30 19:17:34.899482 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.899318    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-credential-socket\") pod \"111205d6-93ca-400e-8383-1b1c555d47dc\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") "
Oct 30 19:17:34.899482 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.899364    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111205d6-93ca-400e-8383-1b1c555d47dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111205d6-93ca-400e-8383-1b1c555d47dc-istiod-ca-cert\") pod \"111205d6-93ca-400e-8383-1b1c555d47dc\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") "
Oct 30 19:17:34.899588 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.899559    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-certs\") pod \"111205d6-93ca-400e-8383-1b1c555d47dc\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") "
Oct 30 19:17:34.899642 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.899603    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-data\") pod \"111205d6-93ca-400e-8383-1b1c555d47dc\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") "
Oct 30 19:17:34.899693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.899637    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-envoy\") pod \"111205d6-93ca-400e-8383-1b1c555d47dc\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") "
Oct 30 19:17:34.899693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.899655    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-socket" pod "111205d6-93ca-400e-8383-1b1c555d47dc" (UID: "111205d6-93ca-400e-8383-1b1c555d47dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:34.899693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.899680    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/111205d6-93ca-400e-8383-1b1c555d47dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111205d6-93ca-400e-8383-1b1c555d47dc-istio-podinfo\") pod \"111205d6-93ca-400e-8383-1b1c555d47dc\" (UID: \"111205d6-93ca-400e-8383-1b1c555d47dc\") "
Oct 30 19:17:34.900839 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.900038    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111205d6-93ca-400e-8383-1b1c555d47dc-istiod-ca-cert" pod "111205d6-93ca-400e-8383-1b1c555d47dc" (UID: "111205d6-93ca-400e-8383-1b1c555d47dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:34.900839 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.900249    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-credential-socket" pod "111205d6-93ca-400e-8383-1b1c555d47dc" (UID: "111205d6-93ca-400e-8383-1b1c555d47dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:34.900839 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.900420    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-certs" pod "111205d6-93ca-400e-8383-1b1c555d47dc" (UID: "111205d6-93ca-400e-8383-1b1c555d47dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:34.900839 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.900484    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-data" pod "111205d6-93ca-400e-8383-1b1c555d47dc" (UID: "111205d6-93ca-400e-8383-1b1c555d47dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:34.900839 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.900759    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.900839 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.900781    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.900839 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.900796    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.900839 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.900812    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.900839 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.900827    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111205d6-93ca-400e-8383-1b1c555d47dc-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.903732 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.903646    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/111205d6-93ca-400e-8383-1b1c555d47dc-istio-podinfo" pod "111205d6-93ca-400e-8383-1b1c555d47dc" (UID: "111205d6-93ca-400e-8383-1b1c555d47dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:34.904060 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.904019    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-envoy" pod "111205d6-93ca-400e-8383-1b1c555d47dc" (UID: "111205d6-93ca-400e-8383-1b1c555d47dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:34.904060 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.904043    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-istio-token" pod "111205d6-93ca-400e-8383-1b1c555d47dc" (UID: "111205d6-93ca-400e-8383-1b1c555d47dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:34.905579 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.905545    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-kube-api-access-9sr9r" pod "111205d6-93ca-400e-8383-1b1c555d47dc" (UID: "111205d6-93ca-400e-8383-1b1c555d47dc"). InnerVolumeSpecName "kube-api-access-9sr9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:34.930967 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.930193    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"]
Oct 30 19:17:34.977508 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:34.977479    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"]
Oct 30 19:17:34.979025 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:17:34.978989    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1c8c76_4e8d_4808_a8f4_09bd5e49f831.slice/crio-50fbc46ffdfbffa7b698c411f59e09882e6563d72b98319608a2728889c19565 WatchSource:0}: Error finding container 50fbc46ffdfbffa7b698c411f59e09882e6563d72b98319608a2728889c19565: Status 404 returned error can't find the container with id 50fbc46ffdfbffa7b698c411f59e09882e6563d72b98319608a2728889c19565
Oct 30 19:17:35.002285 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:35.002251    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111205d6-93ca-400e-8383-1b1c555d47dc-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:35.002285 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:35.002281    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111205d6-93ca-400e-8383-1b1c555d47dc-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:35.002465 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:35.002297    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sr9r\" (UniqueName: \"kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-kube-api-access-9sr9r\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:35.002465 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:35.002312    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111205d6-93ca-400e-8383-1b1c555d47dc-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:35.476951 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:35.476917    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799cb959-41de-49dd-bab4-d64a2d054208" path="/var/lib/kubelet/pods/799cb959-41de-49dd-bab4-d64a2d054208/volumes"
Oct 30 19:17:35.759093 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:35.759011    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"
Oct 30 19:17:35.759093 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:35.759020    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5" event={"ID":"111205d6-93ca-400e-8383-1b1c555d47dc","Type":"ContainerDied","Data":"9236043218b5cd85596c54098d29cf203a29b18cf92e12504f7ce4fa31c13951"}
Oct 30 19:17:35.759612 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:35.759094    6014 scope.go:122] "RemoveContainer" containerID="434ae9326b9232252ee465690ad10e392ca7c3488607dd8e88d0b4c7cb1d7c8e"
Oct 30 19:17:35.760624 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:35.760598    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx" event={"ID":"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831","Type":"ContainerStarted","Data":"50fbc46ffdfbffa7b698c411f59e09882e6563d72b98319608a2728889c19565"}
Oct 30 19:17:35.923701 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:35.923665    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"]
Oct 30 19:17:35.972975 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:35.972936    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-ssdc5"]
Oct 30 19:17:36.111768 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:36.111729    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb" podUID="cbd18ae9-9e76-42e1-bde3-da20fcc21ad8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:36.772588 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:36.772553    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx" event={"ID":"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831","Type":"ContainerStarted","Data":"8594f100a9197a487e719ee79387a35485c522304fca82f1ab62db05a7f78df8"}
Oct 30 19:17:36.773046 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:36.772778    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:36.888512 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:36.888174    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx" podStartSLOduration=3.084788965 podStartE2EDuration="3.888133693s" totalImagesPullingTime="803.344728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:36.887131034 +0000 UTC m=+2295.530298649" watchObservedRunningTime="2026-10-30 19:17:36.888133693 +0000 UTC m=+2295.531301308"
Oct 30 19:17:37.476820 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:37.476784    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111205d6-93ca-400e-8383-1b1c555d47dc" path="/var/lib/kubelet/pods/111205d6-93ca-400e-8383-1b1c555d47dc/volumes"
Oct 30 19:17:40.928476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:40.928440    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"]
Oct 30 19:17:42.785956 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:42.785928    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:17:45.929511 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:45.929450    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9" podUID="03e372cb-2b3a-48a7-b6ec-fca4e17652a9" containerName="payload-processing" containerID="cri-o://2c4bd3b3fb12e335f7e7e3370454276118a7ee11def3634dde8a725fd292e88d" gracePeriod=25
Oct 30 19:17:46.110976 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:46.110933    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb" podUID="cbd18ae9-9e76-42e1-bde3-da20fcc21ad8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:49.737721 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.737688    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"
Oct 30 19:17:49.832791 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.832757    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-plugins-config-volume\") pod \"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8\" (UID: \"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8\") "
Oct 30 19:17:49.832959 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.832811    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-kube-api-access-54mpn\" (UniqueName: \"kubernetes.io/projected/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-kube-api-access-54mpn\") pod \"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8\" (UID: \"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8\") "
Oct 30 19:17:49.833210 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.833187    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-plugins-config-volume" pod "cbd18ae9-9e76-42e1-bde3-da20fcc21ad8" (UID: "cbd18ae9-9e76-42e1-bde3-da20fcc21ad8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:49.833339 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.833322    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:49.835122 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.835099    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-kube-api-access-54mpn" pod "cbd18ae9-9e76-42e1-bde3-da20fcc21ad8" (UID: "cbd18ae9-9e76-42e1-bde3-da20fcc21ad8"). InnerVolumeSpecName "kube-api-access-54mpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:49.861356 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.861324    6014 generic.go:356] "Generic (PLEG): container finished" podID="cbd18ae9-9e76-42e1-bde3-da20fcc21ad8" containerID="674d30df0aa1a1d439ddee5f2bd20898e0bff436d87d288357e7093515a1f121" exitCode=0
Oct 30 19:17:49.861512 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.861388    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"
Oct 30 19:17:49.861512 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.861411    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb" event={"ID":"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8","Type":"ContainerDied","Data":"674d30df0aa1a1d439ddee5f2bd20898e0bff436d87d288357e7093515a1f121"}
Oct 30 19:17:49.861512 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.861467    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb" event={"ID":"cbd18ae9-9e76-42e1-bde3-da20fcc21ad8","Type":"ContainerDied","Data":"2a575c795b7ee716f20d69997e6153af0876d5f0f1c890eb8b2de5dd1584b196"}
Oct 30 19:17:49.861512 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.861489    6014 scope.go:122] "RemoveContainer" containerID="674d30df0aa1a1d439ddee5f2bd20898e0bff436d87d288357e7093515a1f121"
Oct 30 19:17:49.872446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.872418    6014 scope.go:122] "RemoveContainer" containerID="674d30df0aa1a1d439ddee5f2bd20898e0bff436d87d288357e7093515a1f121"
Oct 30 19:17:49.872820 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:17:49.872789    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674d30df0aa1a1d439ddee5f2bd20898e0bff436d87d288357e7093515a1f121\": container with ID starting with 674d30df0aa1a1d439ddee5f2bd20898e0bff436d87d288357e7093515a1f121 not found: ID does not exist" containerID="674d30df0aa1a1d439ddee5f2bd20898e0bff436d87d288357e7093515a1f121"
Oct 30 19:17:49.872910 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.872850    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674d30df0aa1a1d439ddee5f2bd20898e0bff436d87d288357e7093515a1f121"} err="failed to get container status \"674d30df0aa1a1d439ddee5f2bd20898e0bff436d87d288357e7093515a1f121\": rpc error: code = NotFound desc = could not find container \"674d30df0aa1a1d439ddee5f2bd20898e0bff436d87d288357e7093515a1f121\": container with ID starting with 674d30df0aa1a1d439ddee5f2bd20898e0bff436d87d288357e7093515a1f121 not found: ID does not exist"
Oct 30 19:17:49.934427 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.934396    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54mpn\" (UniqueName: \"kubernetes.io/projected/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8-kube-api-access-54mpn\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:49.978537 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:49.978430    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main" containerID="cri-o://34eeca231dccd2dda7023d5d343c84802dcbf9ade4ddf306e45e33eb2270abed" gracePeriod=45
Oct 30 19:17:50.026067 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.026036    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"]
Oct 30 19:17:50.029859 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.029833    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-41db7c-9c5bdfc7f-kwtjb"]
Oct 30 19:17:50.246575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.246549    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:17:50.338848 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.338823    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/664bad09-b36f-46cd-8970-034c6094bd65-tls-certs\" (UniqueName: \"kubernetes.io/secret/664bad09-b36f-46cd-8970-034c6094bd65-tls-certs\") pod \"664bad09-b36f-46cd-8970-034c6094bd65\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") "
Oct 30 19:17:50.339025 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.338932    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664bad09-b36f-46cd-8970-034c6094bd65-kube-api-access-v5gsq\" (UniqueName: \"kubernetes.io/projected/664bad09-b36f-46cd-8970-034c6094bd65-kube-api-access-v5gsq\") pod \"664bad09-b36f-46cd-8970-034c6094bd65\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") "
Oct 30 19:17:50.339025 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.338994    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-tmp-dir\") pod \"664bad09-b36f-46cd-8970-034c6094bd65\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") "
Oct 30 19:17:50.339199 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.339030    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-model-cache\") pod \"664bad09-b36f-46cd-8970-034c6094bd65\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") "
Oct 30 19:17:50.339199 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.339127    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-dshm\") pod \"664bad09-b36f-46cd-8970-034c6094bd65\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") "
Oct 30 19:17:50.339322 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.339240    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-home\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-home\") pod \"664bad09-b36f-46cd-8970-034c6094bd65\" (UID: \"664bad09-b36f-46cd-8970-034c6094bd65\") "
Oct 30 19:17:50.339520 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.339443    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-model-cache" pod "664bad09-b36f-46cd-8970-034c6094bd65" (UID: "664bad09-b36f-46cd-8970-034c6094bd65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:50.339654 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.339631    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-tmp-dir" pod "664bad09-b36f-46cd-8970-034c6094bd65" (UID: "664bad09-b36f-46cd-8970-034c6094bd65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:50.339954 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.339932    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-home" pod "664bad09-b36f-46cd-8970-034c6094bd65" (UID: "664bad09-b36f-46cd-8970-034c6094bd65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:50.340180 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.340129    6014 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-tmp-dir\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.340256 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.340189    6014 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-model-cache\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.340256 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.340205    6014 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-home\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.341501 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.341478    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/664bad09-b36f-46cd-8970-034c6094bd65-tls-certs" pod "664bad09-b36f-46cd-8970-034c6094bd65" (UID: "664bad09-b36f-46cd-8970-034c6094bd65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:50.341501 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.341481    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664bad09-b36f-46cd-8970-034c6094bd65-kube-api-access-v5gsq" pod "664bad09-b36f-46cd-8970-034c6094bd65" (UID: "664bad09-b36f-46cd-8970-034c6094bd65"). InnerVolumeSpecName "kube-api-access-v5gsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:50.341893 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.341870    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-dshm" pod "664bad09-b36f-46cd-8970-034c6094bd65" (UID: "664bad09-b36f-46cd-8970-034c6094bd65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:50.441620 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.441584    6014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/664bad09-b36f-46cd-8970-034c6094bd65-tls-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.441620 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.441616    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5gsq\" (UniqueName: \"kubernetes.io/projected/664bad09-b36f-46cd-8970-034c6094bd65-kube-api-access-v5gsq\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.441620 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.441626    6014 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/664bad09-b36f-46cd-8970-034c6094bd65-dshm\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.870913 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.870874    6014 generic.go:356] "Generic (PLEG): container finished" podID="664bad09-b36f-46cd-8970-034c6094bd65" containerID="34eeca231dccd2dda7023d5d343c84802dcbf9ade4ddf306e45e33eb2270abed" exitCode=0
Oct 30 19:17:50.870913 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.870920    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"
Oct 30 19:17:50.871545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.870948    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" event={"ID":"664bad09-b36f-46cd-8970-034c6094bd65","Type":"ContainerDied","Data":"34eeca231dccd2dda7023d5d343c84802dcbf9ade4ddf306e45e33eb2270abed"}
Oct 30 19:17:50.871545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.870987    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr" event={"ID":"664bad09-b36f-46cd-8970-034c6094bd65","Type":"ContainerDied","Data":"ec82ec85e401b7455d447afa402e9f824905630ace7bbb42f7d46c69409d4417"}
Oct 30 19:17:50.871545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.871011    6014 scope.go:122] "RemoveContainer" containerID="34eeca231dccd2dda7023d5d343c84802dcbf9ade4ddf306e45e33eb2270abed"
Oct 30 19:17:50.881075 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.881056    6014 scope.go:122] "RemoveContainer" containerID="34eeca231dccd2dda7023d5d343c84802dcbf9ade4ddf306e45e33eb2270abed"
Oct 30 19:17:50.881385 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:17:50.881367    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34eeca231dccd2dda7023d5d343c84802dcbf9ade4ddf306e45e33eb2270abed\": container with ID starting with 34eeca231dccd2dda7023d5d343c84802dcbf9ade4ddf306e45e33eb2270abed not found: ID does not exist" containerID="34eeca231dccd2dda7023d5d343c84802dcbf9ade4ddf306e45e33eb2270abed"
Oct 30 19:17:50.881443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:50.881394    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34eeca231dccd2dda7023d5d343c84802dcbf9ade4ddf306e45e33eb2270abed"} err="failed to get container status \"34eeca231dccd2dda7023d5d343c84802dcbf9ade4ddf306e45e33eb2270abed\": rpc error: code = NotFound desc = could not find container \"34eeca231dccd2dda7023d5d343c84802dcbf9ade4ddf306e45e33eb2270abed\": container with ID starting with 34eeca231dccd2dda7023d5d343c84802dcbf9ade4ddf306e45e33eb2270abed not found: ID does not exist"
Oct 30 19:17:51.035186 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:51.035153    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"]
Oct 30 19:17:51.084595 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:51.084557    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/simulator-w1-975cf5-kserve-6cbc4cc6dc-pqrjr"]
Oct 30 19:17:51.476811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:51.476779    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="664bad09-b36f-46cd-8970-034c6094bd65" path="/var/lib/kubelet/pods/664bad09-b36f-46cd-8970-034c6094bd65/volumes"
Oct 30 19:17:51.477217 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:51.477204    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd18ae9-9e76-42e1-bde3-da20fcc21ad8" path="/var/lib/kubelet/pods/cbd18ae9-9e76-42e1-bde3-da20fcc21ad8/volumes"
Oct 30 19:17:53.197268 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:53.197237    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9" podUID="03e372cb-2b3a-48a7-b6ec-fca4e17652a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:53.259000 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:17:53.258967    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:17:53.459177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:53.459058    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"]
Oct 30 19:17:53.571906 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:53.571876    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"]
Oct 30 19:17:57.140844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:57.140811    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:17:58.459558 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:58.459516    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6" podUID="f3c98a0a-4137-408a-9811-3a27f4866a84" containerName="payload-processing" containerID="cri-o://42493e5fe9cb498d1288b2b6ee0ff317f72aab8d8cb56627f1102358b0fcafef" gracePeriod=25
Oct 30 19:17:58.572311 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:17:58.572239    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6" podUID="68730cc9-c11b-4699-a58e-792d376fe6d0" containerName="payload-pre-processing" containerID="cri-o://e588ade265079af3558ec4a0d8892e30c59d618bb09992ed03048bbbb27b4083" gracePeriod=25
Oct 30 19:18:01.084718 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:01.084672    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6" podUID="f3c98a0a-4137-408a-9811-3a27f4866a84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:18:01.085129 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:01.084726    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6" podUID="68730cc9-c11b-4699-a58e-792d376fe6d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:18:03.197696 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:03.197656    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9" podUID="03e372cb-2b3a-48a7-b6ec-fca4e17652a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:18:06.146179 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:06.145257    6014 generic.go:356] "Generic (PLEG): container finished" podID="03e372cb-2b3a-48a7-b6ec-fca4e17652a9" containerID="2c4bd3b3fb12e335f7e7e3370454276118a7ee11def3634dde8a725fd292e88d" exitCode=0
Oct 30 19:18:06.146179 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:06.145294    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9" event={"ID":"03e372cb-2b3a-48a7-b6ec-fca4e17652a9","Type":"ContainerDied","Data":"2c4bd3b3fb12e335f7e7e3370454276118a7ee11def3634dde8a725fd292e88d"}
Oct 30 19:18:06.231559 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:06.231528    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"
Oct 30 19:18:06.351164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:06.346618    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-kube-api-access-lz95x\" (UniqueName: \"kubernetes.io/projected/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-kube-api-access-lz95x\") pod \"03e372cb-2b3a-48a7-b6ec-fca4e17652a9\" (UID: \"03e372cb-2b3a-48a7-b6ec-fca4e17652a9\") "
Oct 30 19:18:06.351164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:06.346847    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-plugins-config-volume\") pod \"03e372cb-2b3a-48a7-b6ec-fca4e17652a9\" (UID: \"03e372cb-2b3a-48a7-b6ec-fca4e17652a9\") "
Oct 30 19:18:06.351164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:06.347696    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-plugins-config-volume" pod "03e372cb-2b3a-48a7-b6ec-fca4e17652a9" (UID: "03e372cb-2b3a-48a7-b6ec-fca4e17652a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:06.352159 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:06.352091    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-kube-api-access-lz95x" pod "03e372cb-2b3a-48a7-b6ec-fca4e17652a9" (UID: "03e372cb-2b3a-48a7-b6ec-fca4e17652a9"). InnerVolumeSpecName "kube-api-access-lz95x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:06.449619 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:06.449564    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:06.449619 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:06.449617    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz95x\" (UniqueName: \"kubernetes.io/projected/03e372cb-2b3a-48a7-b6ec-fca4e17652a9-kube-api-access-lz95x\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:07.153011 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:07.152971    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"
Oct 30 19:18:07.153011 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:07.153002    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9" event={"ID":"03e372cb-2b3a-48a7-b6ec-fca4e17652a9","Type":"ContainerDied","Data":"ff229af754b2b9fd58663a4303a36fe262789b85f0b28d1b7da69ecc3641ea88"}
Oct 30 19:18:07.153635 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:07.153068    6014 scope.go:122] "RemoveContainer" containerID="2c4bd3b3fb12e335f7e7e3370454276118a7ee11def3634dde8a725fd292e88d"
Oct 30 19:18:07.316886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:07.316836    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"]
Oct 30 19:18:07.320657 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:07.320625    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-975cf5-6b9685f66c-f8zn9"]
Oct 30 19:18:07.477027 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:07.476939    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e372cb-2b3a-48a7-b6ec-fca4e17652a9" path="/var/lib/kubelet/pods/03e372cb-2b3a-48a7-b6ec-fca4e17652a9/volumes"
Oct 30 19:18:09.170686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:09.170652    6014 generic.go:356] "Generic (PLEG): container finished" podID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerID="253907f23b82ed9f9d85af5cc97549680fdbaf6b8b9d5fde2abfd5bb23ff6bf7" exitCode=0
Oct 30 19:18:09.171300 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:09.170734    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9" event={"ID":"cd2baa7c-faac-4276-b89e-2b16917d3495","Type":"ContainerDied","Data":"253907f23b82ed9f9d85af5cc97549680fdbaf6b8b9d5fde2abfd5bb23ff6bf7"}
Oct 30 19:18:09.171397 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:09.171338    6014 scope.go:122] "RemoveContainer" containerID="253907f23b82ed9f9d85af5cc97549680fdbaf6b8b9d5fde2abfd5bb23ff6bf7"
Oct 30 19:18:09.172964 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:09.172937    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:18:09.173031 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:09.173005    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:18:09.173069 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:09.173038    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:18:10.179421 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:10.179375    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9" event={"ID":"cd2baa7c-faac-4276-b89e-2b16917d3495","Type":"ContainerStarted","Data":"d0e12575132ca27aa7dacde8999847ca451cba4646e5b1e0093b1cec125310f0"}
Oct 30 19:18:11.072166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:11.071631    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:18:11.072166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:11.071772    6014 patch_prober.go:29] interesting pod/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 30 19:18:11.072166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:11.071819    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9" podUID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 30 19:18:11.081701 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:11.081662    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6" podUID="68730cc9-c11b-4699-a58e-792d376fe6d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:18:11.081701 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:11.081687    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6" podUID="f3c98a0a-4137-408a-9811-3a27f4866a84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:18:11.543812 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:11.543686    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"]
Oct 30 19:18:11.544950 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:11.544904    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" podUID="c9723156-3755-4457-8694-8ef4a234d191" containerName="istio-proxy" containerID="cri-o://49057df9c5d6601da560d86fa625a273e9062bd6ae9036b6d7f59cb022340e80" gracePeriod=30
Oct 30 19:18:12.070315 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:12.070273    6014 patch_prober.go:29] interesting pod/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 30 19:18:12.070626 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:12.070594    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9" podUID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 30 19:18:12.992163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:12.992109    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"]
Oct 30 19:18:12.992669 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:12.992388    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv" podUID="855f3c98-2ab1-47ad-883e-85410ae82a77" containerName="maas-api" containerID="cri-o://6343e632ee23b290d9349d6398d09dddba9f97a1a2df53dd0caf28ffcf3a8603" gracePeriod=30
Oct 30 19:18:13.079000 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.078499    6014 patch_prober.go:29] interesting pod/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 30 19:18:13.079000 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.078596    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9" podUID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 30 19:18:13.247172 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.246771    6014 generic.go:356] "Generic (PLEG): container finished" podID="855f3c98-2ab1-47ad-883e-85410ae82a77" containerID="6343e632ee23b290d9349d6398d09dddba9f97a1a2df53dd0caf28ffcf3a8603" exitCode=0
Oct 30 19:18:13.247172 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.247020    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv" event={"ID":"855f3c98-2ab1-47ad-883e-85410ae82a77","Type":"ContainerDied","Data":"6343e632ee23b290d9349d6398d09dddba9f97a1a2df53dd0caf28ffcf3a8603"}
Oct 30 19:18:13.336434 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.336407    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:18:13.440458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.440418    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/855f3c98-2ab1-47ad-883e-85410ae82a77-kube-api-access-q7tdp\" (UniqueName: \"kubernetes.io/projected/855f3c98-2ab1-47ad-883e-85410ae82a77-kube-api-access-q7tdp\") pod \"855f3c98-2ab1-47ad-883e-85410ae82a77\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") "
Oct 30 19:18:13.440650 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.440619    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-maas-api-tls\") pod \"855f3c98-2ab1-47ad-883e-85410ae82a77\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") "
Oct 30 19:18:13.441154 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.441102    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/855f3c98-2ab1-47ad-883e-85410ae82a77-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/855f3c98-2ab1-47ad-883e-85410ae82a77-openshift-service-ca\") pod \"855f3c98-2ab1-47ad-883e-85410ae82a77\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") "
Oct 30 19:18:13.442486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.442095    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-metrics-certs\" (UniqueName: \"kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-metrics-certs\") pod \"855f3c98-2ab1-47ad-883e-85410ae82a77\" (UID: \"855f3c98-2ab1-47ad-883e-85410ae82a77\") "
Oct 30 19:18:13.442486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.442097    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/855f3c98-2ab1-47ad-883e-85410ae82a77-openshift-service-ca" pod "855f3c98-2ab1-47ad-883e-85410ae82a77" (UID: "855f3c98-2ab1-47ad-883e-85410ae82a77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:13.442886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.442859    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/855f3c98-2ab1-47ad-883e-85410ae82a77-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:13.443660 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.443610    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-maas-api-tls" pod "855f3c98-2ab1-47ad-883e-85410ae82a77" (UID: "855f3c98-2ab1-47ad-883e-85410ae82a77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:13.443843 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.443815    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855f3c98-2ab1-47ad-883e-85410ae82a77-kube-api-access-q7tdp" pod "855f3c98-2ab1-47ad-883e-85410ae82a77" (UID: "855f3c98-2ab1-47ad-883e-85410ae82a77"). InnerVolumeSpecName "kube-api-access-q7tdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:13.445044 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.445027    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-metrics-certs" pod "855f3c98-2ab1-47ad-883e-85410ae82a77" (UID: "855f3c98-2ab1-47ad-883e-85410ae82a77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:13.544105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.544076    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7tdp\" (UniqueName: \"kubernetes.io/projected/855f3c98-2ab1-47ad-883e-85410ae82a77-kube-api-access-q7tdp\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:13.544105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.544103    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:13.544444 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:13.544118    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/855f3c98-2ab1-47ad-883e-85410ae82a77-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:14.066170 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:14.064746    6014 patch_prober.go:29] interesting pod/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 30 19:18:14.066170 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:14.064816    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9" podUID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 30 19:18:14.278402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:14.278367    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"
Oct 30 19:18:14.278587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:14.278489    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv" event={"ID":"855f3c98-2ab1-47ad-883e-85410ae82a77","Type":"ContainerDied","Data":"f0c378b4ab850873f5207f1c16512ec75436eba2edad42bd4f295ea12d0f80c7"}
Oct 30 19:18:14.278587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:14.278569    6014 scope.go:122] "RemoveContainer" containerID="6343e632ee23b290d9349d6398d09dddba9f97a1a2df53dd0caf28ffcf3a8603"
Oct 30 19:18:14.471227 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:14.470770    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"]
Oct 30 19:18:14.502676 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:14.502631    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-903a82-5d6794879b-xs8vv"]
Oct 30 19:18:15.067162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:15.066783    6014 patch_prober.go:29] interesting pod/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 30 19:18:15.067162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:15.066888    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9" podUID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 30 19:18:15.128166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:15.106292    6014 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:18:15.128166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:15.106361    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" podUID="c9723156-3755-4457-8694-8ef4a234d191" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:18:15.479908 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:15.479825    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="855f3c98-2ab1-47ad-883e-85410ae82a77" path="/var/lib/kubelet/pods/855f3c98-2ab1-47ad-883e-85410ae82a77/volumes"
Oct 30 19:18:16.056476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.056428    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:18:16.192174 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.192110    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:18:16.300020 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.299989    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:18:16.837297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.837271    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:18:16.912037 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.911930    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-credential-socket\") pod \"c9723156-3755-4457-8694-8ef4a234d191\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") "
Oct 30 19:18:16.912037 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.912023    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-certs\") pod \"c9723156-3755-4457-8694-8ef4a234d191\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") "
Oct 30 19:18:16.912331 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.912074    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-envoy\") pod \"c9723156-3755-4457-8694-8ef4a234d191\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") "
Oct 30 19:18:16.912331 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.912111    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-data\") pod \"c9723156-3755-4457-8694-8ef4a234d191\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") "
Oct 30 19:18:16.912331 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.912209    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-istio-token\" (UniqueName: \"kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-istio-token\") pod \"c9723156-3755-4457-8694-8ef4a234d191\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") "
Oct 30 19:18:16.912331 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.912258    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9723156-3755-4457-8694-8ef4a234d191-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9723156-3755-4457-8694-8ef4a234d191-istiod-ca-cert\") pod \"c9723156-3755-4457-8694-8ef4a234d191\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") "
Oct 30 19:18:16.912331 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.912316    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-socket\") pod \"c9723156-3755-4457-8694-8ef4a234d191\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") "
Oct 30 19:18:16.912703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.912373    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9723156-3755-4457-8694-8ef4a234d191-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9723156-3755-4457-8694-8ef4a234d191-istio-podinfo\") pod \"c9723156-3755-4457-8694-8ef4a234d191\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") "
Oct 30 19:18:16.912703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.912412    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-kube-api-access-rmsg8\" (UniqueName: \"kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-kube-api-access-rmsg8\") pod \"c9723156-3755-4457-8694-8ef4a234d191\" (UID: \"c9723156-3755-4457-8694-8ef4a234d191\") "
Oct 30 19:18:16.912703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.912310    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-credential-socket" pod "c9723156-3755-4457-8694-8ef4a234d191" (UID: "c9723156-3755-4457-8694-8ef4a234d191"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:16.912703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.912431    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-certs" pod "c9723156-3755-4457-8694-8ef4a234d191" (UID: "c9723156-3755-4457-8694-8ef4a234d191"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:16.912703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.912486    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-data" pod "c9723156-3755-4457-8694-8ef4a234d191" (UID: "c9723156-3755-4457-8694-8ef4a234d191"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:16.912703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.912588    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-socket" pod "c9723156-3755-4457-8694-8ef4a234d191" (UID: "c9723156-3755-4457-8694-8ef4a234d191"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:16.912703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.912678    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9723156-3755-4457-8694-8ef4a234d191-istiod-ca-cert" pod "c9723156-3755-4457-8694-8ef4a234d191" (UID: "c9723156-3755-4457-8694-8ef4a234d191"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:16.913544 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.913521    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9723156-3755-4457-8694-8ef4a234d191-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:16.913697 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.913549    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:16.913697 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.913579    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:16.913697 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.913594    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:16.913697 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.913607    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:16.914675 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.914644    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9723156-3755-4457-8694-8ef4a234d191-istio-podinfo" pod "c9723156-3755-4457-8694-8ef4a234d191" (UID: "c9723156-3755-4457-8694-8ef4a234d191"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:18:16.914786 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.914683    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-envoy" pod "c9723156-3755-4457-8694-8ef4a234d191" (UID: "c9723156-3755-4457-8694-8ef4a234d191"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:16.914786 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.914684    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-istio-token" pod "c9723156-3755-4457-8694-8ef4a234d191" (UID: "c9723156-3755-4457-8694-8ef4a234d191"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:16.914867 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:16.914831    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-kube-api-access-rmsg8" pod "c9723156-3755-4457-8694-8ef4a234d191" (UID: "c9723156-3755-4457-8694-8ef4a234d191"). InnerVolumeSpecName "kube-api-access-rmsg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:17.015396 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:17.015358    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9723156-3755-4457-8694-8ef4a234d191-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:17.015396 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:17.015392    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:17.015396 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:17.015405    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9723156-3755-4457-8694-8ef4a234d191-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:17.015636 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:17.015415    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmsg8\" (UniqueName: \"kubernetes.io/projected/c9723156-3755-4457-8694-8ef4a234d191-kube-api-access-rmsg8\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:17.306897 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:17.306802    6014 generic.go:356] "Generic (PLEG): container finished" podID="c9723156-3755-4457-8694-8ef4a234d191" containerID="49057df9c5d6601da560d86fa625a273e9062bd6ae9036b6d7f59cb022340e80" exitCode=0
Oct 30 19:18:17.306897 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:17.306856    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"
Oct 30 19:18:17.307474 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:17.306883    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" event={"ID":"c9723156-3755-4457-8694-8ef4a234d191","Type":"ContainerDied","Data":"49057df9c5d6601da560d86fa625a273e9062bd6ae9036b6d7f59cb022340e80"}
Oct 30 19:18:17.307474 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:17.306952    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x" event={"ID":"c9723156-3755-4457-8694-8ef4a234d191","Type":"ContainerDied","Data":"5ce1494968f5dfe2bb50e52d92398f2048066608f13a79570286b778ef598f7b"}
Oct 30 19:18:17.307474 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:17.306979    6014 scope.go:122] "RemoveContainer" containerID="49057df9c5d6601da560d86fa625a273e9062bd6ae9036b6d7f59cb022340e80"
Oct 30 19:18:17.320853 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:17.320827    6014 scope.go:122] "RemoveContainer" containerID="49057df9c5d6601da560d86fa625a273e9062bd6ae9036b6d7f59cb022340e80"
Oct 30 19:18:17.321610 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:17.321548    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49057df9c5d6601da560d86fa625a273e9062bd6ae9036b6d7f59cb022340e80\": container with ID starting with 49057df9c5d6601da560d86fa625a273e9062bd6ae9036b6d7f59cb022340e80 not found: ID does not exist" containerID="49057df9c5d6601da560d86fa625a273e9062bd6ae9036b6d7f59cb022340e80"
Oct 30 19:18:17.321610 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:17.321578    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49057df9c5d6601da560d86fa625a273e9062bd6ae9036b6d7f59cb022340e80"} err="failed to get container status \"49057df9c5d6601da560d86fa625a273e9062bd6ae9036b6d7f59cb022340e80\": rpc error: code = NotFound desc = could not find container \"49057df9c5d6601da560d86fa625a273e9062bd6ae9036b6d7f59cb022340e80\": container with ID starting with 49057df9c5d6601da560d86fa625a273e9062bd6ae9036b6d7f59cb022340e80 not found: ID does not exist"
Oct 30 19:18:17.477885 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:17.477852    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"]
Oct 30 19:18:17.521882 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:17.521851    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-f7q9x"]
Oct 30 19:18:18.755151 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:18.755116    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"
Oct 30 19:18:18.844090 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:18.844064    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3c98a0a-4137-408a-9811-3a27f4866a84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c98a0a-4137-408a-9811-3a27f4866a84-plugins-config-volume\") pod \"f3c98a0a-4137-408a-9811-3a27f4866a84\" (UID: \"f3c98a0a-4137-408a-9811-3a27f4866a84\") "
Oct 30 19:18:18.844240 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:18.844134    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c98a0a-4137-408a-9811-3a27f4866a84-kube-api-access-w6lph\" (UniqueName: \"kubernetes.io/projected/f3c98a0a-4137-408a-9811-3a27f4866a84-kube-api-access-w6lph\") pod \"f3c98a0a-4137-408a-9811-3a27f4866a84\" (UID: \"f3c98a0a-4137-408a-9811-3a27f4866a84\") "
Oct 30 19:18:18.844496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:18.844471    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c98a0a-4137-408a-9811-3a27f4866a84-plugins-config-volume" pod "f3c98a0a-4137-408a-9811-3a27f4866a84" (UID: "f3c98a0a-4137-408a-9811-3a27f4866a84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:18.844824 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:18.844807    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c98a0a-4137-408a-9811-3a27f4866a84-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:18.846558 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:18.846536    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c98a0a-4137-408a-9811-3a27f4866a84-kube-api-access-w6lph" pod "f3c98a0a-4137-408a-9811-3a27f4866a84" (UID: "f3c98a0a-4137-408a-9811-3a27f4866a84"). InnerVolumeSpecName "kube-api-access-w6lph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:18.853384 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:18.853363    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"
Oct 30 19:18:18.945889 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:18.945858    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68730cc9-c11b-4699-a58e-792d376fe6d0-kube-api-access-sq9vf\" (UniqueName: \"kubernetes.io/projected/68730cc9-c11b-4699-a58e-792d376fe6d0-kube-api-access-sq9vf\") pod \"68730cc9-c11b-4699-a58e-792d376fe6d0\" (UID: \"68730cc9-c11b-4699-a58e-792d376fe6d0\") "
Oct 30 19:18:18.946072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:18.945921    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68730cc9-c11b-4699-a58e-792d376fe6d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68730cc9-c11b-4699-a58e-792d376fe6d0-plugins-config-volume\") pod \"68730cc9-c11b-4699-a58e-792d376fe6d0\" (UID: \"68730cc9-c11b-4699-a58e-792d376fe6d0\") "
Oct 30 19:18:18.946324 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:18.946297    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68730cc9-c11b-4699-a58e-792d376fe6d0-plugins-config-volume" pod "68730cc9-c11b-4699-a58e-792d376fe6d0" (UID: "68730cc9-c11b-4699-a58e-792d376fe6d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:18.946498 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:18.946483    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68730cc9-c11b-4699-a58e-792d376fe6d0-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:18.946543 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:18.946501    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6lph\" (UniqueName: \"kubernetes.io/projected/f3c98a0a-4137-408a-9811-3a27f4866a84-kube-api-access-w6lph\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:18.948108 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:18.948091    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68730cc9-c11b-4699-a58e-792d376fe6d0-kube-api-access-sq9vf" pod "68730cc9-c11b-4699-a58e-792d376fe6d0" (UID: "68730cc9-c11b-4699-a58e-792d376fe6d0"). InnerVolumeSpecName "kube-api-access-sq9vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:19.048220 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.048094    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq9vf\" (UniqueName: \"kubernetes.io/projected/68730cc9-c11b-4699-a58e-792d376fe6d0-kube-api-access-sq9vf\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:19.322572 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.322534    6014 generic.go:356] "Generic (PLEG): container finished" podID="f3c98a0a-4137-408a-9811-3a27f4866a84" containerID="42493e5fe9cb498d1288b2b6ee0ff317f72aab8d8cb56627f1102358b0fcafef" exitCode=0
Oct 30 19:18:19.322789 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.322608    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"
Oct 30 19:18:19.322789 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.322621    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6" event={"ID":"f3c98a0a-4137-408a-9811-3a27f4866a84","Type":"ContainerDied","Data":"42493e5fe9cb498d1288b2b6ee0ff317f72aab8d8cb56627f1102358b0fcafef"}
Oct 30 19:18:19.322789 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.322715    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6" event={"ID":"f3c98a0a-4137-408a-9811-3a27f4866a84","Type":"ContainerDied","Data":"103a225b51f6431cc5bd86195263de5a336cc7dc051ba43ccc88edff351af7ad"}
Oct 30 19:18:19.322789 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.322750    6014 scope.go:122] "RemoveContainer" containerID="42493e5fe9cb498d1288b2b6ee0ff317f72aab8d8cb56627f1102358b0fcafef"
Oct 30 19:18:19.324262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.324239    6014 generic.go:356] "Generic (PLEG): container finished" podID="68730cc9-c11b-4699-a58e-792d376fe6d0" containerID="e588ade265079af3558ec4a0d8892e30c59d618bb09992ed03048bbbb27b4083" exitCode=0
Oct 30 19:18:19.324371 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.324308    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"
Oct 30 19:18:19.324371 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.324317    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6" event={"ID":"68730cc9-c11b-4699-a58e-792d376fe6d0","Type":"ContainerDied","Data":"e588ade265079af3558ec4a0d8892e30c59d618bb09992ed03048bbbb27b4083"}
Oct 30 19:18:19.324486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.324375    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6" event={"ID":"68730cc9-c11b-4699-a58e-792d376fe6d0","Type":"ContainerDied","Data":"83d202ffb0a90c1d771218060668b333de05f1a3919aea922118738ecb880aaf"}
Oct 30 19:18:19.333956 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.333938    6014 scope.go:122] "RemoveContainer" containerID="42493e5fe9cb498d1288b2b6ee0ff317f72aab8d8cb56627f1102358b0fcafef"
Oct 30 19:18:19.334325 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:19.334299    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42493e5fe9cb498d1288b2b6ee0ff317f72aab8d8cb56627f1102358b0fcafef\": container with ID starting with 42493e5fe9cb498d1288b2b6ee0ff317f72aab8d8cb56627f1102358b0fcafef not found: ID does not exist" containerID="42493e5fe9cb498d1288b2b6ee0ff317f72aab8d8cb56627f1102358b0fcafef"
Oct 30 19:18:19.334408 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.334339    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42493e5fe9cb498d1288b2b6ee0ff317f72aab8d8cb56627f1102358b0fcafef"} err="failed to get container status \"42493e5fe9cb498d1288b2b6ee0ff317f72aab8d8cb56627f1102358b0fcafef\": rpc error: code = NotFound desc = could not find container \"42493e5fe9cb498d1288b2b6ee0ff317f72aab8d8cb56627f1102358b0fcafef\": container with ID starting with 42493e5fe9cb498d1288b2b6ee0ff317f72aab8d8cb56627f1102358b0fcafef not found: ID does not exist"
Oct 30 19:18:19.334408 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.334365    6014 scope.go:122] "RemoveContainer" containerID="e588ade265079af3558ec4a0d8892e30c59d618bb09992ed03048bbbb27b4083"
Oct 30 19:18:19.344242 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.344222    6014 scope.go:122] "RemoveContainer" containerID="e588ade265079af3558ec4a0d8892e30c59d618bb09992ed03048bbbb27b4083"
Oct 30 19:18:19.344497 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:19.344480    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e588ade265079af3558ec4a0d8892e30c59d618bb09992ed03048bbbb27b4083\": container with ID starting with e588ade265079af3558ec4a0d8892e30c59d618bb09992ed03048bbbb27b4083 not found: ID does not exist" containerID="e588ade265079af3558ec4a0d8892e30c59d618bb09992ed03048bbbb27b4083"
Oct 30 19:18:19.344553 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.344504    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e588ade265079af3558ec4a0d8892e30c59d618bb09992ed03048bbbb27b4083"} err="failed to get container status \"e588ade265079af3558ec4a0d8892e30c59d618bb09992ed03048bbbb27b4083\": rpc error: code = NotFound desc = could not find container \"e588ade265079af3558ec4a0d8892e30c59d618bb09992ed03048bbbb27b4083\": container with ID starting with e588ade265079af3558ec4a0d8892e30c59d618bb09992ed03048bbbb27b4083 not found: ID does not exist"
Oct 30 19:18:19.442312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.442278    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"]
Oct 30 19:18:19.443121 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443099    6014 cpu_manager.go:414] "containerMap: removing container" podUID="68730cc9-c11b-4699-a58e-792d376fe6d0" containerName="payload-pre-processing"
Oct 30 19:18:19.443121 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443121    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68730cc9-c11b-4699-a58e-792d376fe6d0" containerName="payload-pre-processing"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443134    6014 cpu_manager.go:414] "containerMap: removing container" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443154    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443162    6014 cpu_manager.go:414] "containerMap: removing container" podUID="c9723156-3755-4457-8694-8ef4a234d191" containerName="istio-proxy"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443167    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9723156-3755-4457-8694-8ef4a234d191" containerName="istio-proxy"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443180    6014 cpu_manager.go:414] "containerMap: removing container" podUID="855f3c98-2ab1-47ad-883e-85410ae82a77" containerName="maas-api"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443185    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="855f3c98-2ab1-47ad-883e-85410ae82a77" containerName="maas-api"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443210    6014 cpu_manager.go:414] "containerMap: removing container" podUID="111205d6-93ca-400e-8383-1b1c555d47dc" containerName="istio-proxy"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443215    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111205d6-93ca-400e-8383-1b1c555d47dc" containerName="istio-proxy"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443222    6014 cpu_manager.go:414] "containerMap: removing container" podUID="f3c98a0a-4137-408a-9811-3a27f4866a84" containerName="payload-processing"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443227    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c98a0a-4137-408a-9811-3a27f4866a84" containerName="payload-processing"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443235    6014 cpu_manager.go:414] "containerMap: removing container" podUID="152f91b3-d4d9-4cf2-a821-12062ad0fb9b" containerName="revoke-keys"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443239    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="152f91b3-d4d9-4cf2-a821-12062ad0fb9b" containerName="revoke-keys"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443246    6014 cpu_manager.go:414] "containerMap: removing container" podUID="03e372cb-2b3a-48a7-b6ec-fca4e17652a9" containerName="payload-processing"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443250    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e372cb-2b3a-48a7-b6ec-fca4e17652a9" containerName="payload-processing"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443258    6014 cpu_manager.go:414] "containerMap: removing container" podUID="cbd18ae9-9e76-42e1-bde3-da20fcc21ad8" containerName="payload-pre-processing"
Oct 30 19:18:19.443301 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443262    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd18ae9-9e76-42e1-bde3-da20fcc21ad8" containerName="payload-pre-processing"
Oct 30 19:18:19.443833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443361    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="111205d6-93ca-400e-8383-1b1c555d47dc" containerName="istio-proxy"
Oct 30 19:18:19.443833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443369    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c98a0a-4137-408a-9811-3a27f4866a84" containerName="payload-processing"
Oct 30 19:18:19.443833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443378    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="152f91b3-d4d9-4cf2-a821-12062ad0fb9b" containerName="revoke-keys"
Oct 30 19:18:19.443833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443392    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e372cb-2b3a-48a7-b6ec-fca4e17652a9" containerName="payload-processing"
Oct 30 19:18:19.443833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443398    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd18ae9-9e76-42e1-bde3-da20fcc21ad8" containerName="payload-pre-processing"
Oct 30 19:18:19.443833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443409    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9723156-3755-4457-8694-8ef4a234d191" containerName="istio-proxy"
Oct 30 19:18:19.443833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443415    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="68730cc9-c11b-4699-a58e-792d376fe6d0" containerName="payload-pre-processing"
Oct 30 19:18:19.443833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443423    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="664bad09-b36f-46cd-8970-034c6094bd65" containerName="main"
Oct 30 19:18:19.443833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.443430    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="855f3c98-2ab1-47ad-883e-85410ae82a77" containerName="maas-api"
Oct 30 19:18:19.446727 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.446702    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"
Oct 30 19:18:19.479038 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.479012    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9723156-3755-4457-8694-8ef4a234d191" path="/var/lib/kubelet/pods/c9723156-3755-4457-8694-8ef4a234d191/volumes"
Oct 30 19:18:19.489580 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.489534    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"]
Oct 30 19:18:19.492067 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.492041    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1290a6-7b774bb9f-vnpt6"]
Oct 30 19:18:19.502445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.502418    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"]
Oct 30 19:18:19.545348 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.545316    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:18:19.545509 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.545408    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:18:19.555200 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.555169    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vmsn\" (UniqueName: \"kubernetes.io/projected/e0e84da3-cc5f-493f-8aa5-2d566333062f-kube-api-access-4vmsn\") pod \"maas-api-revoke-sub-86875e66-dcrn4\" (UID: \"e0e84da3-cc5f-493f-8aa5-2d566333062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"
Oct 30 19:18:19.555419 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.555402    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86875e66-dcrn4\" (UID: \"e0e84da3-cc5f-493f-8aa5-2d566333062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"
Oct 30 19:18:19.555540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.555521    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86875e66-dcrn4\" (UID: \"e0e84da3-cc5f-493f-8aa5-2d566333062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"
Oct 30 19:18:19.599338 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.599245    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"]
Oct 30 19:18:19.605200 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.605168    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1290a6-b56b64589-kbnx6"]
Oct 30 19:18:19.657239 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.657198    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86875e66-dcrn4\" (UID: \"e0e84da3-cc5f-493f-8aa5-2d566333062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"
Oct 30 19:18:19.657447 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.657352    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86875e66-dcrn4\" (UID: \"e0e84da3-cc5f-493f-8aa5-2d566333062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"
Oct 30 19:18:19.657529 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.657501    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vmsn\" (UniqueName: \"kubernetes.io/projected/e0e84da3-cc5f-493f-8aa5-2d566333062f-kube-api-access-4vmsn\") pod \"maas-api-revoke-sub-86875e66-dcrn4\" (UID: \"e0e84da3-cc5f-493f-8aa5-2d566333062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"
Oct 30 19:18:19.658084 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.658057    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86875e66-dcrn4\" (UID: \"e0e84da3-cc5f-493f-8aa5-2d566333062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"
Oct 30 19:18:19.659844 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.659815    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86875e66-dcrn4\" (UID: \"e0e84da3-cc5f-493f-8aa5-2d566333062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"
Oct 30 19:18:19.716412 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.716379    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vmsn\" (UniqueName: \"kubernetes.io/projected/e0e84da3-cc5f-493f-8aa5-2d566333062f-kube-api-access-4vmsn\") pod \"maas-api-revoke-sub-86875e66-dcrn4\" (UID: \"e0e84da3-cc5f-493f-8aa5-2d566333062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"
Oct 30 19:18:19.760177 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:19.760127    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"
Oct 30 19:18:20.135419 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:20.135392    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"]
Oct 30 19:18:20.137551 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:18:20.137525    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e84da3_cc5f_493f_8aa5_2d566333062f.slice/crio-94d0e54cdcdf450ae3a4d735d5796e3afd07128f9d6b225fed9eb4a6e43327b7 WatchSource:0}: Error finding container 94d0e54cdcdf450ae3a4d735d5796e3afd07128f9d6b225fed9eb4a6e43327b7: Status 404 returned error can't find the container with id 94d0e54cdcdf450ae3a4d735d5796e3afd07128f9d6b225fed9eb4a6e43327b7
Oct 30 19:18:20.332244 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:20.332201    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4" event={"ID":"e0e84da3-cc5f-493f-8aa5-2d566333062f","Type":"ContainerStarted","Data":"94d0e54cdcdf450ae3a4d735d5796e3afd07128f9d6b225fed9eb4a6e43327b7"}
Oct 30 19:18:21.342296 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:21.342259    6014 generic.go:356] "Generic (PLEG): container finished" podID="e0e84da3-cc5f-493f-8aa5-2d566333062f" containerID="0a5af15919b60336e29b194314907258f3f86f751dd3fbf05361084c81e65a58" exitCode=0
Oct 30 19:18:21.342715 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:21.342339    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4" event={"ID":"e0e84da3-cc5f-493f-8aa5-2d566333062f","Type":"ContainerDied","Data":"0a5af15919b60336e29b194314907258f3f86f751dd3fbf05361084c81e65a58"}
Oct 30 19:18:21.476623 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:21.476549    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68730cc9-c11b-4699-a58e-792d376fe6d0" path="/var/lib/kubelet/pods/68730cc9-c11b-4699-a58e-792d376fe6d0/volumes"
Oct 30 19:18:21.476986 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:21.476972    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3c98a0a-4137-408a-9811-3a27f4866a84" path="/var/lib/kubelet/pods/f3c98a0a-4137-408a-9811-3a27f4866a84/volumes"
Oct 30 19:18:21.992335 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:21.992302    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"]
Oct 30 19:18:21.992572 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:21.992547    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h" podUID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerName="istio-proxy" containerID="cri-o://e68ef00a0d664948bb263145a4c5db8deea3a2b1737b86f1421fc1801e32d2c0" gracePeriod=30
Oct 30 19:18:22.505398 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.505370    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"
Oct 30 19:18:22.603908 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.603830    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-openshift-ca\") pod \"e0e84da3-cc5f-493f-8aa5-2d566333062f\" (UID: \"e0e84da3-cc5f-493f-8aa5-2d566333062f\") "
Oct 30 19:18:22.603908 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.603885    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-xks-ca\") pod \"e0e84da3-cc5f-493f-8aa5-2d566333062f\" (UID: \"e0e84da3-cc5f-493f-8aa5-2d566333062f\") "
Oct 30 19:18:22.604113 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.604037    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e84da3-cc5f-493f-8aa5-2d566333062f-kube-api-access-4vmsn\" (UniqueName: \"kubernetes.io/projected/e0e84da3-cc5f-493f-8aa5-2d566333062f-kube-api-access-4vmsn\") pod \"e0e84da3-cc5f-493f-8aa5-2d566333062f\" (UID: \"e0e84da3-cc5f-493f-8aa5-2d566333062f\") "
Oct 30 19:18:22.604258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.604235    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-openshift-ca" pod "e0e84da3-cc5f-493f-8aa5-2d566333062f" (UID: "e0e84da3-cc5f-493f-8aa5-2d566333062f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:22.604680 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.604611    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:22.606337 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.606310    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-xks-ca" pod "e0e84da3-cc5f-493f-8aa5-2d566333062f" (UID: "e0e84da3-cc5f-493f-8aa5-2d566333062f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:22.606454 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.606387    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e84da3-cc5f-493f-8aa5-2d566333062f-kube-api-access-4vmsn" pod "e0e84da3-cc5f-493f-8aa5-2d566333062f" (UID: "e0e84da3-cc5f-493f-8aa5-2d566333062f"). InnerVolumeSpecName "kube-api-access-4vmsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:22.705877 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.705842    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e84da3-cc5f-493f-8aa5-2d566333062f-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:22.705877 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.705874    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vmsn\" (UniqueName: \"kubernetes.io/projected/e0e84da3-cc5f-493f-8aa5-2d566333062f-kube-api-access-4vmsn\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:22.707643 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.707618    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"]
Oct 30 19:18:22.708317 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.708299    6014 cpu_manager.go:414] "containerMap: removing container" podUID="e0e84da3-cc5f-493f-8aa5-2d566333062f" containerName="revoke-keys"
Oct 30 19:18:22.708379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.708319    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e84da3-cc5f-493f-8aa5-2d566333062f" containerName="revoke-keys"
Oct 30 19:18:22.708547 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.708531    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e84da3-cc5f-493f-8aa5-2d566333062f" containerName="revoke-keys"
Oct 30 19:18:22.710859 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.710845    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"
Oct 30 19:18:22.770058 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.770020    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"]
Oct 30 19:18:22.807044 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.807009    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcxgk\" (UniqueName: \"kubernetes.io/projected/8897a808-dd04-489e-96f6-f5506ebe5d5a-kube-api-access-gcxgk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57\" (UID: \"8897a808-dd04-489e-96f6-f5506ebe5d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"
Oct 30 19:18:22.807262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.807159    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57\" (UID: \"8897a808-dd04-489e-96f6-f5506ebe5d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"
Oct 30 19:18:22.807262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.807238    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57\" (UID: \"8897a808-dd04-489e-96f6-f5506ebe5d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"
Oct 30 19:18:22.908104 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.908016    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcxgk\" (UniqueName: \"kubernetes.io/projected/8897a808-dd04-489e-96f6-f5506ebe5d5a-kube-api-access-gcxgk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57\" (UID: \"8897a808-dd04-489e-96f6-f5506ebe5d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"
Oct 30 19:18:22.908104 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.908096    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57\" (UID: \"8897a808-dd04-489e-96f6-f5506ebe5d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"
Oct 30 19:18:22.908402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.908128    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57\" (UID: \"8897a808-dd04-489e-96f6-f5506ebe5d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"
Oct 30 19:18:22.908889 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.908863    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57\" (UID: \"8897a808-dd04-489e-96f6-f5506ebe5d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"
Oct 30 19:18:22.910806 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.910781    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57\" (UID: \"8897a808-dd04-489e-96f6-f5506ebe5d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"
Oct 30 19:18:22.993543 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:22.993501    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcxgk\" (UniqueName: \"kubernetes.io/projected/8897a808-dd04-489e-96f6-f5506ebe5d5a-kube-api-access-gcxgk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57\" (UID: \"8897a808-dd04-489e-96f6-f5506ebe5d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"
Oct 30 19:18:23.024025 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:23.023986    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"
Oct 30 19:18:23.262510 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:23.262131    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:18:23.358216 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:23.358177    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4" event={"ID":"e0e84da3-cc5f-493f-8aa5-2d566333062f","Type":"ContainerDied","Data":"94d0e54cdcdf450ae3a4d735d5796e3afd07128f9d6b225fed9eb4a6e43327b7"}
Oct 30 19:18:23.358216 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:23.358213    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"
Oct 30 19:18:23.358446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:23.358240    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94d0e54cdcdf450ae3a4d735d5796e3afd07128f9d6b225fed9eb4a6e43327b7"
Oct 30 19:18:23.424678 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:23.424641    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"]
Oct 30 19:18:24.367437 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:24.367398    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57" event={"ID":"8897a808-dd04-489e-96f6-f5506ebe5d5a","Type":"ContainerStarted","Data":"4b2e25c6a5b5c69fa72010a00a153511989c7ba12154d43e5af1b1995cae1887"}
Oct 30 19:18:24.367875 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:24.367459    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57" event={"ID":"8897a808-dd04-489e-96f6-f5506ebe5d5a","Type":"ContainerStarted","Data":"0c5d7a83bfc8176b4489175080939baba62c12224a5652fa5bbc534997dc6b1e"}
Oct 30 19:18:24.490426 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:24.490372    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57" podStartSLOduration=2.490354808 podStartE2EDuration="2.490354808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:24.487771051 +0000 UTC m=+2343.130938666" watchObservedRunningTime="2026-10-30 19:18:24.490354808 +0000 UTC m=+2343.133522421"
Oct 30 19:18:25.380333 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:25.380291    6014 generic.go:356] "Generic (PLEG): container finished" podID="8897a808-dd04-489e-96f6-f5506ebe5d5a" containerID="4b2e25c6a5b5c69fa72010a00a153511989c7ba12154d43e5af1b1995cae1887" exitCode=0
Oct 30 19:18:25.380833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:25.380345    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57" event={"ID":"8897a808-dd04-489e-96f6-f5506ebe5d5a","Type":"ContainerDied","Data":"4b2e25c6a5b5c69fa72010a00a153511989c7ba12154d43e5af1b1995cae1887"}
Oct 30 19:18:26.545749 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:26.545720    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"
Oct 30 19:18:26.661258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:26.661168    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-openshift-ca\") pod \"8897a808-dd04-489e-96f6-f5506ebe5d5a\" (UID: \"8897a808-dd04-489e-96f6-f5506ebe5d5a\") "
Oct 30 19:18:26.661443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:26.661303    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-xks-ca\") pod \"8897a808-dd04-489e-96f6-f5506ebe5d5a\" (UID: \"8897a808-dd04-489e-96f6-f5506ebe5d5a\") "
Oct 30 19:18:26.661443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:26.661429    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8897a808-dd04-489e-96f6-f5506ebe5d5a-kube-api-access-gcxgk\" (UniqueName: \"kubernetes.io/projected/8897a808-dd04-489e-96f6-f5506ebe5d5a-kube-api-access-gcxgk\") pod \"8897a808-dd04-489e-96f6-f5506ebe5d5a\" (UID: \"8897a808-dd04-489e-96f6-f5506ebe5d5a\") "
Oct 30 19:18:26.661633 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:26.661604    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-openshift-ca" pod "8897a808-dd04-489e-96f6-f5506ebe5d5a" (UID: "8897a808-dd04-489e-96f6-f5506ebe5d5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:26.662022 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:26.661993    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:26.664488 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:26.664449    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-xks-ca" pod "8897a808-dd04-489e-96f6-f5506ebe5d5a" (UID: "8897a808-dd04-489e-96f6-f5506ebe5d5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:26.664590 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:26.664555    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8897a808-dd04-489e-96f6-f5506ebe5d5a-kube-api-access-gcxgk" pod "8897a808-dd04-489e-96f6-f5506ebe5d5a" (UID: "8897a808-dd04-489e-96f6-f5506ebe5d5a"). InnerVolumeSpecName "kube-api-access-gcxgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:26.763455 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:26.763415    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcxgk\" (UniqueName: \"kubernetes.io/projected/8897a808-dd04-489e-96f6-f5506ebe5d5a-kube-api-access-gcxgk\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:26.763455 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:26.763449    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8897a808-dd04-489e-96f6-f5506ebe5d5a-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:27.248809 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.248783    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:18:27.270581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.270493    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-podinfo\") pod \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") "
Oct 30 19:18:27.270581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.270537    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-certs\") pod \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") "
Oct 30 19:18:27.270581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.270580    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-kube-api-access-xddjn\" (UniqueName: \"kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-kube-api-access-xddjn\") pod \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") "
Oct 30 19:18:27.270845 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.270607    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-token\" (UniqueName: \"kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-token\") pod \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") "
Oct 30 19:18:27.270845 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.270796    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-socket\") pod \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") "
Oct 30 19:18:27.270965 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.270846    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istiod-ca-cert\") pod \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") "
Oct 30 19:18:27.270965 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.270891    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-certs" pod "e693b661-2b77-4bf5-82f7-b11fd1e570bb" (UID: "e693b661-2b77-4bf5-82f7-b11fd1e570bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:27.271133 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.271110    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-socket" pod "e693b661-2b77-4bf5-82f7-b11fd1e570bb" (UID: "e693b661-2b77-4bf5-82f7-b11fd1e570bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:27.271235 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.270942    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-envoy\") pod \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") "
Oct 30 19:18:27.271287 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.271261    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istiod-ca-cert" pod "e693b661-2b77-4bf5-82f7-b11fd1e570bb" (UID: "e693b661-2b77-4bf5-82f7-b11fd1e570bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:27.271347 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.271338    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-credential-socket\") pod \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") "
Oct 30 19:18:27.271403 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.271367    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-data\") pod \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\" (UID: \"e693b661-2b77-4bf5-82f7-b11fd1e570bb\") "
Oct 30 19:18:27.271698 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.271675    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-credential-socket" pod "e693b661-2b77-4bf5-82f7-b11fd1e570bb" (UID: "e693b661-2b77-4bf5-82f7-b11fd1e570bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:27.271877 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.271850    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-data" pod "e693b661-2b77-4bf5-82f7-b11fd1e570bb" (UID: "e693b661-2b77-4bf5-82f7-b11fd1e570bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:27.272043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.272028    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:27.272095 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.272045    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:27.272095 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.272056    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:27.272095 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.272065    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:27.272095 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.272073    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:27.273555 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.273523    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-podinfo" pod "e693b661-2b77-4bf5-82f7-b11fd1e570bb" (UID: "e693b661-2b77-4bf5-82f7-b11fd1e570bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:18:27.273662 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.273619    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-kube-api-access-xddjn" pod "e693b661-2b77-4bf5-82f7-b11fd1e570bb" (UID: "e693b661-2b77-4bf5-82f7-b11fd1e570bb"). InnerVolumeSpecName "kube-api-access-xddjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:27.273731 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.273691    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-envoy" pod "e693b661-2b77-4bf5-82f7-b11fd1e570bb" (UID: "e693b661-2b77-4bf5-82f7-b11fd1e570bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:27.273731 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.273708    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-token" pod "e693b661-2b77-4bf5-82f7-b11fd1e570bb" (UID: "e693b661-2b77-4bf5-82f7-b11fd1e570bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:27.373551 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.373515    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:27.373551 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.373543    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:27.373551 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.373553    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xddjn\" (UniqueName: \"kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-kube-api-access-xddjn\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:27.373551 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.373562    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e693b661-2b77-4bf5-82f7-b11fd1e570bb-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:27.394597 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.394567    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"
Oct 30 19:18:27.394804 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.394557    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57" event={"ID":"8897a808-dd04-489e-96f6-f5506ebe5d5a","Type":"ContainerDied","Data":"0c5d7a83bfc8176b4489175080939baba62c12224a5652fa5bbc534997dc6b1e"}
Oct 30 19:18:27.394804 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.394701    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c5d7a83bfc8176b4489175080939baba62c12224a5652fa5bbc534997dc6b1e"
Oct 30 19:18:27.396387 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.396366    6014 generic.go:356] "Generic (PLEG): container finished" podID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerID="e68ef00a0d664948bb263145a4c5db8deea3a2b1737b86f1421fc1801e32d2c0" exitCode=0
Oct 30 19:18:27.396548 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.396396    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h" event={"ID":"e693b661-2b77-4bf5-82f7-b11fd1e570bb","Type":"ContainerDied","Data":"e68ef00a0d664948bb263145a4c5db8deea3a2b1737b86f1421fc1801e32d2c0"}
Oct 30 19:18:27.396548 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.396428    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h" event={"ID":"e693b661-2b77-4bf5-82f7-b11fd1e570bb","Type":"ContainerDied","Data":"b020b88c02170cc2dff750a0002252a1b92250c7922f86344057bf5f01ef971b"}
Oct 30 19:18:27.396548 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.396429    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"
Oct 30 19:18:27.396548 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.396458    6014 scope.go:122] "RemoveContainer" containerID="e68ef00a0d664948bb263145a4c5db8deea3a2b1737b86f1421fc1801e32d2c0"
Oct 30 19:18:27.407630 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.407612    6014 scope.go:122] "RemoveContainer" containerID="62696aee7734892c47050c55661aaac6d53cdba08b03020d45c2355cace1640d"
Oct 30 19:18:27.418718 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.418675    6014 scope.go:122] "RemoveContainer" containerID="e68ef00a0d664948bb263145a4c5db8deea3a2b1737b86f1421fc1801e32d2c0"
Oct 30 19:18:27.419062 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:27.419036    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68ef00a0d664948bb263145a4c5db8deea3a2b1737b86f1421fc1801e32d2c0\": container with ID starting with e68ef00a0d664948bb263145a4c5db8deea3a2b1737b86f1421fc1801e32d2c0 not found: ID does not exist" containerID="e68ef00a0d664948bb263145a4c5db8deea3a2b1737b86f1421fc1801e32d2c0"
Oct 30 19:18:27.419345 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.419074    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68ef00a0d664948bb263145a4c5db8deea3a2b1737b86f1421fc1801e32d2c0"} err="failed to get container status \"e68ef00a0d664948bb263145a4c5db8deea3a2b1737b86f1421fc1801e32d2c0\": rpc error: code = NotFound desc = could not find container \"e68ef00a0d664948bb263145a4c5db8deea3a2b1737b86f1421fc1801e32d2c0\": container with ID starting with e68ef00a0d664948bb263145a4c5db8deea3a2b1737b86f1421fc1801e32d2c0 not found: ID does not exist"
Oct 30 19:18:27.419345 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.419103    6014 scope.go:122] "RemoveContainer" containerID="62696aee7734892c47050c55661aaac6d53cdba08b03020d45c2355cace1640d"
Oct 30 19:18:27.419630 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:27.419603    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62696aee7734892c47050c55661aaac6d53cdba08b03020d45c2355cace1640d\": container with ID starting with 62696aee7734892c47050c55661aaac6d53cdba08b03020d45c2355cace1640d not found: ID does not exist" containerID="62696aee7734892c47050c55661aaac6d53cdba08b03020d45c2355cace1640d"
Oct 30 19:18:27.419727 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.419635    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62696aee7734892c47050c55661aaac6d53cdba08b03020d45c2355cace1640d"} err="failed to get container status \"62696aee7734892c47050c55661aaac6d53cdba08b03020d45c2355cace1640d\": rpc error: code = NotFound desc = could not find container \"62696aee7734892c47050c55661aaac6d53cdba08b03020d45c2355cace1640d\": container with ID starting with 62696aee7734892c47050c55661aaac6d53cdba08b03020d45c2355cace1640d not found: ID does not exist"
Oct 30 19:18:27.628212 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.628176    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"]
Oct 30 19:18:27.633192 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:27.633158    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-903a82-openshift-default-6474b78d8f-fxg7h"]
Oct 30 19:18:29.476784 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:29.476750    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" path="/var/lib/kubelet/pods/e693b661-2b77-4bf5-82f7-b11fd1e570bb/volumes"
Oct 30 19:18:30.104573 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.104542    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"]
Oct 30 19:18:30.105212 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.105198    6014 cpu_manager.go:414] "containerMap: removing container" podUID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerName="istio-proxy"
Oct 30 19:18:30.105271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.105214    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerName="istio-proxy"
Oct 30 19:18:30.105271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.105239    6014 cpu_manager.go:414] "containerMap: removing container" podUID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerName="istio-proxy"
Oct 30 19:18:30.105271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.105244    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerName="istio-proxy"
Oct 30 19:18:30.105271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.105267    6014 cpu_manager.go:414] "containerMap: removing container" podUID="8897a808-dd04-489e-96f6-f5506ebe5d5a" containerName="revoke-keys"
Oct 30 19:18:30.105395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.105272    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8897a808-dd04-489e-96f6-f5506ebe5d5a" containerName="revoke-keys"
Oct 30 19:18:30.105395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.105356    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerName="istio-proxy"
Oct 30 19:18:30.105395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.105368    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="8897a808-dd04-489e-96f6-f5506ebe5d5a" containerName="revoke-keys"
Oct 30 19:18:30.110402 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.110378    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"
Oct 30 19:18:30.166354 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.166319    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"]
Oct 30 19:18:30.209574 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.209537    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97c1fc0e-9pp9h\" (UID: \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"
Oct 30 19:18:30.209796 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.209711    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97c1fc0e-9pp9h\" (UID: \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"
Oct 30 19:18:30.209796 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.209760    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5kp\" (UniqueName: \"kubernetes.io/projected/9ea8cb62-852f-4bd1-b948-218ec0b214e6-kube-api-access-lj5kp\") pod \"maas-api-revoke-sub-97c1fc0e-9pp9h\" (UID: \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"
Oct 30 19:18:30.211394 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.211355    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:18:30.211394 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.211367    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:18:30.311738 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.311696    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97c1fc0e-9pp9h\" (UID: \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"
Oct 30 19:18:30.311938 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.311760    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5kp\" (UniqueName: \"kubernetes.io/projected/9ea8cb62-852f-4bd1-b948-218ec0b214e6-kube-api-access-lj5kp\") pod \"maas-api-revoke-sub-97c1fc0e-9pp9h\" (UID: \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"
Oct 30 19:18:30.311938 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.311895    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97c1fc0e-9pp9h\" (UID: \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"
Oct 30 19:18:30.312425 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.312405    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97c1fc0e-9pp9h\" (UID: \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"
Oct 30 19:18:30.314516 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.314489    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97c1fc0e-9pp9h\" (UID: \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"
Oct 30 19:18:30.369507 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.369428    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5kp\" (UniqueName: \"kubernetes.io/projected/9ea8cb62-852f-4bd1-b948-218ec0b214e6-kube-api-access-lj5kp\") pod \"maas-api-revoke-sub-97c1fc0e-9pp9h\" (UID: \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"
Oct 30 19:18:30.430724 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.430638    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"
Oct 30 19:18:30.811501 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:30.811476    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"]
Oct 30 19:18:30.813453 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:18:30.813423    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea8cb62_852f_4bd1_b948_218ec0b214e6.slice/crio-f5bbfc222f83aab48a4efc82f0078c95349f7f0df442c94c2c133792165076ba WatchSource:0}: Error finding container f5bbfc222f83aab48a4efc82f0078c95349f7f0df442c94c2c133792165076ba: Status 404 returned error can't find the container with id f5bbfc222f83aab48a4efc82f0078c95349f7f0df442c94c2c133792165076ba
Oct 30 19:18:31.424852 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:31.424818    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h" event={"ID":"9ea8cb62-852f-4bd1-b948-218ec0b214e6","Type":"ContainerStarted","Data":"194ac092019fb7cb8f6596f238599cd0aeb167e825b0cf5bfa65dbaea0db2af4"}
Oct 30 19:18:31.425113 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:31.424869    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h" event={"ID":"9ea8cb62-852f-4bd1-b948-218ec0b214e6","Type":"ContainerStarted","Data":"f5bbfc222f83aab48a4efc82f0078c95349f7f0df442c94c2c133792165076ba"}
Oct 30 19:18:31.544105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:31.544039    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h" podStartSLOduration=1.544019062 podStartE2EDuration="1.544019062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:31.541746041 +0000 UTC m=+2350.184913655" watchObservedRunningTime="2026-10-30 19:18:31.544019062 +0000 UTC m=+2350.187186677"
Oct 30 19:18:32.432752 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:32.432719    6014 generic.go:356] "Generic (PLEG): container finished" podID="9ea8cb62-852f-4bd1-b948-218ec0b214e6" containerID="194ac092019fb7cb8f6596f238599cd0aeb167e825b0cf5bfa65dbaea0db2af4" exitCode=0
Oct 30 19:18:32.433286 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:32.432804    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h" event={"ID":"9ea8cb62-852f-4bd1-b948-218ec0b214e6","Type":"ContainerDied","Data":"194ac092019fb7cb8f6596f238599cd0aeb167e825b0cf5bfa65dbaea0db2af4"}
Oct 30 19:18:33.247954 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:33.247913    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:18:33.601868 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:33.601842    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"
Oct 30 19:18:33.657133 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:33.657100    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-xks-ca\") pod \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\" (UID: \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\") "
Oct 30 19:18:33.657322 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:33.657257    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea8cb62-852f-4bd1-b948-218ec0b214e6-kube-api-access-lj5kp\" (UniqueName: \"kubernetes.io/projected/9ea8cb62-852f-4bd1-b948-218ec0b214e6-kube-api-access-lj5kp\") pod \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\" (UID: \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\") "
Oct 30 19:18:33.657398 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:33.657359    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-openshift-ca\") pod \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\" (UID: \"9ea8cb62-852f-4bd1-b948-218ec0b214e6\") "
Oct 30 19:18:33.657723 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:33.657693    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-openshift-ca" pod "9ea8cb62-852f-4bd1-b948-218ec0b214e6" (UID: "9ea8cb62-852f-4bd1-b948-218ec0b214e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:33.658038 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:33.658015    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:33.659931 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:33.659872    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-xks-ca" pod "9ea8cb62-852f-4bd1-b948-218ec0b214e6" (UID: "9ea8cb62-852f-4bd1-b948-218ec0b214e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:33.660037 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:33.660001    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea8cb62-852f-4bd1-b948-218ec0b214e6-kube-api-access-lj5kp" pod "9ea8cb62-852f-4bd1-b948-218ec0b214e6" (UID: "9ea8cb62-852f-4bd1-b948-218ec0b214e6"). InnerVolumeSpecName "kube-api-access-lj5kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:33.759201 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:33.759172    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj5kp\" (UniqueName: \"kubernetes.io/projected/9ea8cb62-852f-4bd1-b948-218ec0b214e6-kube-api-access-lj5kp\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:33.759201 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:33.759198    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea8cb62-852f-4bd1-b948-218ec0b214e6-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:34.453162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:34.453065    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h" event={"ID":"9ea8cb62-852f-4bd1-b948-218ec0b214e6","Type":"ContainerDied","Data":"f5bbfc222f83aab48a4efc82f0078c95349f7f0df442c94c2c133792165076ba"}
Oct 30 19:18:34.453448 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:34.453429    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5bbfc222f83aab48a4efc82f0078c95349f7f0df442c94c2c133792165076ba"
Oct 30 19:18:34.453611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:34.453600    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"
Oct 30 19:18:43.900422 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:43.900381    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"]
Oct 30 19:18:43.901117 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:43.900739    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerName="istio-proxy" containerID="cri-o://a2b42546592e0b62e2796816651022c71baec4de51e3a1548d433fd47cf9c28e" gracePeriod=30
Oct 30 19:18:43.949520 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:43.949480    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"]
Oct 30 19:18:43.949731 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:43.949695    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g" podUID="d328b8d4-3583-4984-9435-1cb4349ae6aa" containerName="istio-proxy" containerID="cri-o://3c25dc92195ebeb056395c6e2d42da5031f7bf7c36f8af5e53888eac34d29280" gracePeriod=30
Oct 30 19:18:46.011564 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:46.011531    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"]
Oct 30 19:18:46.012022 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:46.011744    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm" podUID="da67831f-c12e-47db-ae2b-6a9276287b74" containerName="istio-proxy" containerID="cri-o://739b0abed625d038576d8792f4e8152f235b9dfa0a015f92e7fba509bc3a5346" gracePeriod=30
Oct 30 19:18:46.031858 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:46.031826    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"]
Oct 30 19:18:46.032062 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:46.032042    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerName="istio-proxy" containerID="cri-o://46283fd697700f9f52b70067fbcda31ca3db86b06fe0ead3384fbcc5b3df205f" gracePeriod=30
Oct 30 19:18:46.056879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:46.056845    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"]
Oct 30 19:18:46.057074 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:46.057052    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t" podUID="f84ead0c-c769-4f58-bc0f-e5d7528ae907" containerName="istio-proxy" containerID="cri-o://345454db6c78678434492d6cbdd97ce4008e8bfb856c85b5118776a3662d2f53" gracePeriod=30
Oct 30 19:18:48.029947 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.029907    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"]
Oct 30 19:18:48.030700 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.030679    6014 cpu_manager.go:414] "containerMap: removing container" podUID="9ea8cb62-852f-4bd1-b948-218ec0b214e6" containerName="revoke-keys"
Oct 30 19:18:48.030700 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.030698    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea8cb62-852f-4bd1-b948-218ec0b214e6" containerName="revoke-keys"
Oct 30 19:18:48.031234 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.030842    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea8cb62-852f-4bd1-b948-218ec0b214e6" containerName="revoke-keys"
Oct 30 19:18:48.031234 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.030873    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e693b661-2b77-4bf5-82f7-b11fd1e570bb" containerName="istio-proxy"
Oct 30 19:18:48.034329 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.034312    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.096421 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.096377    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tls-certs\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.096630 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.096442    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tmp-dir\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.096753 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.096722    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-dshm\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.096826 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.096801    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8rq4\" (UniqueName: \"kubernetes.io/projected/da64e353-542e-4e5a-8a31-ef6c1891e5f8-kube-api-access-b8rq4\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.096891 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.096852    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-model-cache\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.096978 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.096957    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-home\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.116928 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.116886    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e0fc1f\"/\"kube-root-ca.crt\""
Oct 30 19:18:48.137489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.137454    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e0fc1f\"/\"openshift-service-ca.crt\""
Oct 30 19:18:48.137698 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.137533    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e0fc1f\"/\"default-dockercfg-glqxz\""
Oct 30 19:18:48.137698 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.137640    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e0fc1f\"/\"test-model-e0fc1f-kserve-self-signed-certs\""
Oct 30 19:18:48.140299 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.140277    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"]
Oct 30 19:18:48.198854 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.198784    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tmp-dir\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.199962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.199244    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-dshm\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.199962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.199319    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8rq4\" (UniqueName: \"kubernetes.io/projected/da64e353-542e-4e5a-8a31-ef6c1891e5f8-kube-api-access-b8rq4\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.199962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.199363    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-model-cache\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.199962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.199403    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-home\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.199962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.199557    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tls-certs\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.203928 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.203897    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tls-certs\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.204431 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.204392    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tmp-dir\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.210219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.208426    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-model-cache\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.210219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.208884    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-dshm\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.210219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.209256    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-home\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.264495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.264457    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8rq4\" (UniqueName: \"kubernetes.io/projected/da64e353-542e-4e5a-8a31-ef6c1891e5f8-kube-api-access-b8rq4\") pod \"test-model-e0fc1f-kserve-b797b84f-xxbll\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") " pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.347359 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.347329    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:48.731902 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:48.731872    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"]
Oct 30 19:18:48.733643 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:18:48.733616    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda64e353_542e_4e5a_8a31_ef6c1891e5f8.slice/crio-135650686ee46c9e99a237c08386e4cf71d8941ae49c6a95a5dc7902fdddf23d WatchSource:0}: Error finding container 135650686ee46c9e99a237c08386e4cf71d8941ae49c6a95a5dc7902fdddf23d: Status 404 returned error can't find the container with id 135650686ee46c9e99a237c08386e4cf71d8941ae49c6a95a5dc7902fdddf23d
Oct 30 19:18:49.049389 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.049309    6014 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 19:18:49.049389 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.049361    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 19:18:49.208801 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.208764    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 19:18:49.274549 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.274525    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:18:49.318716 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.318689    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-kube-api-access-l4q59\" (UniqueName: \"kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-kube-api-access-l4q59\") pod \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") "
Oct 30 19:18:49.318889 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.318728    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-credential-socket\") pod \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") "
Oct 30 19:18:49.318889 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.318787    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-envoy\") pod \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") "
Oct 30 19:18:49.318889 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.318826    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-data\") pod \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") "
Oct 30 19:18:49.319012 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.318895    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istiod-ca-cert\") pod \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") "
Oct 30 19:18:49.319012 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.318953    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-token\" (UniqueName: \"kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-token\") pod \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") "
Oct 30 19:18:49.319121 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.319009    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-podinfo\") pod \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") "
Oct 30 19:18:49.319121 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.319049    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-socket\") pod \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") "
Oct 30 19:18:49.319121 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.319089    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-certs\") pod \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\" (UID: \"a13d3d6b-d787-4b0e-a46a-4a132174f16f\") "
Oct 30 19:18:49.319121 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.319065    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-credential-socket" pod "a13d3d6b-d787-4b0e-a46a-4a132174f16f" (UID: "a13d3d6b-d787-4b0e-a46a-4a132174f16f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:49.319447 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.319171    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-data" pod "a13d3d6b-d787-4b0e-a46a-4a132174f16f" (UID: "a13d3d6b-d787-4b0e-a46a-4a132174f16f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:49.319447 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.319347    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istiod-ca-cert" pod "a13d3d6b-d787-4b0e-a46a-4a132174f16f" (UID: "a13d3d6b-d787-4b0e-a46a-4a132174f16f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:49.319447 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.319410    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-socket" pod "a13d3d6b-d787-4b0e-a46a-4a132174f16f" (UID: "a13d3d6b-d787-4b0e-a46a-4a132174f16f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:49.319629 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.319536    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-certs" pod "a13d3d6b-d787-4b0e-a46a-4a132174f16f" (UID: "a13d3d6b-d787-4b0e-a46a-4a132174f16f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:49.319778 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.319757    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.319904 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.319781    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.319904 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.319796    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.319904 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.319809    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.319904 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.319821    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.321318 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.321288    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-podinfo" pod "a13d3d6b-d787-4b0e-a46a-4a132174f16f" (UID: "a13d3d6b-d787-4b0e-a46a-4a132174f16f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:18:49.321457 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.321437    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-token" pod "a13d3d6b-d787-4b0e-a46a-4a132174f16f" (UID: "a13d3d6b-d787-4b0e-a46a-4a132174f16f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:49.321695 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.321674    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-envoy" pod "a13d3d6b-d787-4b0e-a46a-4a132174f16f" (UID: "a13d3d6b-d787-4b0e-a46a-4a132174f16f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:49.321793 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.321701    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-kube-api-access-l4q59" pod "a13d3d6b-d787-4b0e-a46a-4a132174f16f" (UID: "a13d3d6b-d787-4b0e-a46a-4a132174f16f"). InnerVolumeSpecName "kube-api-access-l4q59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:49.420233 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420202    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-data\") pod \"d328b8d4-3583-4984-9435-1cb4349ae6aa\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") "
Oct 30 19:18:49.420451 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420254    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-envoy\") pod \"d328b8d4-3583-4984-9435-1cb4349ae6aa\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") "
Oct 30 19:18:49.420451 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420294    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-credential-socket\") pod \"d328b8d4-3583-4984-9435-1cb4349ae6aa\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") "
Oct 30 19:18:49.420451 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420337    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d328b8d4-3583-4984-9435-1cb4349ae6aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d328b8d4-3583-4984-9435-1cb4349ae6aa-istiod-ca-cert\") pod \"d328b8d4-3583-4984-9435-1cb4349ae6aa\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") "
Oct 30 19:18:49.420451 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420448    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-certs\") pod \"d328b8d4-3583-4984-9435-1cb4349ae6aa\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") "
Oct 30 19:18:49.420709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420485    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-token\" (UniqueName: \"kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-token\") pod \"d328b8d4-3583-4984-9435-1cb4349ae6aa\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") "
Oct 30 19:18:49.420709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420521    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-data" pod "d328b8d4-3583-4984-9435-1cb4349ae6aa" (UID: "d328b8d4-3583-4984-9435-1cb4349ae6aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:49.420709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420535    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-socket\") pod \"d328b8d4-3583-4984-9435-1cb4349ae6aa\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") "
Oct 30 19:18:49.420709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420613    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-credential-socket" pod "d328b8d4-3583-4984-9435-1cb4349ae6aa" (UID: "d328b8d4-3583-4984-9435-1cb4349ae6aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:49.420709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420695    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-kube-api-access-6hsnd\" (UniqueName: \"kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-kube-api-access-6hsnd\") pod \"d328b8d4-3583-4984-9435-1cb4349ae6aa\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") "
Oct 30 19:18:49.420889 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420709    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-certs" pod "d328b8d4-3583-4984-9435-1cb4349ae6aa" (UID: "d328b8d4-3583-4984-9435-1cb4349ae6aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:49.420889 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420727    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-socket" pod "d328b8d4-3583-4984-9435-1cb4349ae6aa" (UID: "d328b8d4-3583-4984-9435-1cb4349ae6aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:49.420889 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420782    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-podinfo\") pod \"d328b8d4-3583-4984-9435-1cb4349ae6aa\" (UID: \"d328b8d4-3583-4984-9435-1cb4349ae6aa\") "
Oct 30 19:18:49.420889 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.420796    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d328b8d4-3583-4984-9435-1cb4349ae6aa-istiod-ca-cert" pod "d328b8d4-3583-4984-9435-1cb4349ae6aa" (UID: "d328b8d4-3583-4984-9435-1cb4349ae6aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:49.421637 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.421607    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.421775 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.421640    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.421775 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.421656    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4q59\" (UniqueName: \"kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-kube-api-access-l4q59\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.421775 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.421672    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.421775 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.421687    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.421775 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.421705    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.421775 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.421719    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.421775 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.421732    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d328b8d4-3583-4984-9435-1cb4349ae6aa-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.421775 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.421748    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13d3d6b-d787-4b0e-a46a-4a132174f16f-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.422847 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.422819    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-token" pod "d328b8d4-3583-4984-9435-1cb4349ae6aa" (UID: "d328b8d4-3583-4984-9435-1cb4349ae6aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:49.422973 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.422880    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-envoy" pod "d328b8d4-3583-4984-9435-1cb4349ae6aa" (UID: "d328b8d4-3583-4984-9435-1cb4349ae6aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:49.422973 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.422933    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-kube-api-access-6hsnd" pod "d328b8d4-3583-4984-9435-1cb4349ae6aa" (UID: "d328b8d4-3583-4984-9435-1cb4349ae6aa"). InnerVolumeSpecName "kube-api-access-6hsnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:49.423083 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.423023    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-podinfo" pod "d328b8d4-3583-4984-9435-1cb4349ae6aa" (UID: "d328b8d4-3583-4984-9435-1cb4349ae6aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:18:49.522970 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.522938    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.522970 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.522967    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.522970 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.522976    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.523224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.522987    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hsnd\" (UniqueName: \"kubernetes.io/projected/d328b8d4-3583-4984-9435-1cb4349ae6aa-kube-api-access-6hsnd\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:49.552254 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.552217    6014 generic.go:356] "Generic (PLEG): container finished" podID="d328b8d4-3583-4984-9435-1cb4349ae6aa" containerID="3c25dc92195ebeb056395c6e2d42da5031f7bf7c36f8af5e53888eac34d29280" exitCode=0
Oct 30 19:18:49.552438 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.552280    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"
Oct 30 19:18:49.552438 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.552296    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g" event={"ID":"d328b8d4-3583-4984-9435-1cb4349ae6aa","Type":"ContainerDied","Data":"3c25dc92195ebeb056395c6e2d42da5031f7bf7c36f8af5e53888eac34d29280"}
Oct 30 19:18:49.552438 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.552363    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g" event={"ID":"d328b8d4-3583-4984-9435-1cb4349ae6aa","Type":"ContainerDied","Data":"4225eb29b45a57f51df064d8ee74ec91d5ae1e43c0f8d2814c73e748bf0b8ca3"}
Oct 30 19:18:49.552438 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.552393    6014 scope.go:122] "RemoveContainer" containerID="3c25dc92195ebeb056395c6e2d42da5031f7bf7c36f8af5e53888eac34d29280"
Oct 30 19:18:49.554746 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.554724    6014 generic.go:356] "Generic (PLEG): container finished" podID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerID="a2b42546592e0b62e2796816651022c71baec4de51e3a1548d433fd47cf9c28e" exitCode=0
Oct 30 19:18:49.554873 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.554860    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" event={"ID":"a13d3d6b-d787-4b0e-a46a-4a132174f16f","Type":"ContainerDied","Data":"a2b42546592e0b62e2796816651022c71baec4de51e3a1548d433fd47cf9c28e"}
Oct 30 19:18:49.554981 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.554905    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l" event={"ID":"a13d3d6b-d787-4b0e-a46a-4a132174f16f","Type":"ContainerDied","Data":"1db1f7dae32cbd7784dc650213fcd280cadaf321dcd696e067f99f6ae7dd6d9e"}
Oct 30 19:18:49.555048 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.554987    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"
Oct 30 19:18:49.558502 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.558479    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll" event={"ID":"da64e353-542e-4e5a-8a31-ef6c1891e5f8","Type":"ContainerStarted","Data":"0bb09bd015ced379c4ca439b9e3cd0cc645df647da45043b41bcabf5cde2be1b"}
Oct 30 19:18:49.558622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.558514    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll" event={"ID":"da64e353-542e-4e5a-8a31-ef6c1891e5f8","Type":"ContainerStarted","Data":"135650686ee46c9e99a237c08386e4cf71d8941ae49c6a95a5dc7902fdddf23d"}
Oct 30 19:18:49.571834 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.571810    6014 scope.go:122] "RemoveContainer" containerID="3c25dc92195ebeb056395c6e2d42da5031f7bf7c36f8af5e53888eac34d29280"
Oct 30 19:18:49.572210 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:49.572189    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c25dc92195ebeb056395c6e2d42da5031f7bf7c36f8af5e53888eac34d29280\": container with ID starting with 3c25dc92195ebeb056395c6e2d42da5031f7bf7c36f8af5e53888eac34d29280 not found: ID does not exist" containerID="3c25dc92195ebeb056395c6e2d42da5031f7bf7c36f8af5e53888eac34d29280"
Oct 30 19:18:49.572364 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.572222    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c25dc92195ebeb056395c6e2d42da5031f7bf7c36f8af5e53888eac34d29280"} err="failed to get container status \"3c25dc92195ebeb056395c6e2d42da5031f7bf7c36f8af5e53888eac34d29280\": rpc error: code = NotFound desc = could not find container \"3c25dc92195ebeb056395c6e2d42da5031f7bf7c36f8af5e53888eac34d29280\": container with ID starting with 3c25dc92195ebeb056395c6e2d42da5031f7bf7c36f8af5e53888eac34d29280 not found: ID does not exist"
Oct 30 19:18:49.572364 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.572246    6014 scope.go:122] "RemoveContainer" containerID="a2b42546592e0b62e2796816651022c71baec4de51e3a1548d433fd47cf9c28e"
Oct 30 19:18:49.582088 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.582069    6014 scope.go:122] "RemoveContainer" containerID="4ccad808aac5beeb6f8c43f2aa9661459394e46bb09aa63970586304dd693ab7"
Oct 30 19:18:49.591548 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.591525    6014 scope.go:122] "RemoveContainer" containerID="a2b42546592e0b62e2796816651022c71baec4de51e3a1548d433fd47cf9c28e"
Oct 30 19:18:49.591828 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:49.591805    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b42546592e0b62e2796816651022c71baec4de51e3a1548d433fd47cf9c28e\": container with ID starting with a2b42546592e0b62e2796816651022c71baec4de51e3a1548d433fd47cf9c28e not found: ID does not exist" containerID="a2b42546592e0b62e2796816651022c71baec4de51e3a1548d433fd47cf9c28e"
Oct 30 19:18:49.591878 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.591835    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b42546592e0b62e2796816651022c71baec4de51e3a1548d433fd47cf9c28e"} err="failed to get container status \"a2b42546592e0b62e2796816651022c71baec4de51e3a1548d433fd47cf9c28e\": rpc error: code = NotFound desc = could not find container \"a2b42546592e0b62e2796816651022c71baec4de51e3a1548d433fd47cf9c28e\": container with ID starting with a2b42546592e0b62e2796816651022c71baec4de51e3a1548d433fd47cf9c28e not found: ID does not exist"
Oct 30 19:18:49.591878 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.591852    6014 scope.go:122] "RemoveContainer" containerID="4ccad808aac5beeb6f8c43f2aa9661459394e46bb09aa63970586304dd693ab7"
Oct 30 19:18:49.592081 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:49.592062    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccad808aac5beeb6f8c43f2aa9661459394e46bb09aa63970586304dd693ab7\": container with ID starting with 4ccad808aac5beeb6f8c43f2aa9661459394e46bb09aa63970586304dd693ab7 not found: ID does not exist" containerID="4ccad808aac5beeb6f8c43f2aa9661459394e46bb09aa63970586304dd693ab7"
Oct 30 19:18:49.592131 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.592088    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccad808aac5beeb6f8c43f2aa9661459394e46bb09aa63970586304dd693ab7"} err="failed to get container status \"4ccad808aac5beeb6f8c43f2aa9661459394e46bb09aa63970586304dd693ab7\": rpc error: code = NotFound desc = could not find container \"4ccad808aac5beeb6f8c43f2aa9661459394e46bb09aa63970586304dd693ab7\": container with ID starting with 4ccad808aac5beeb6f8c43f2aa9661459394e46bb09aa63970586304dd693ab7 not found: ID does not exist"
Oct 30 19:18:49.716013 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.715979    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"]
Oct 30 19:18:49.721089 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.721064    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-gns5g"]
Oct 30 19:18:49.833388 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.833289    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll" podStartSLOduration=2.833273662 podStartE2EDuration="2.833273662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:49.831334593 +0000 UTC m=+2368.474502208" watchObservedRunningTime="2026-10-30 19:18:49.833273662 +0000 UTC m=+2368.476441276"
Oct 30 19:18:49.951220 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.947494    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"]
Oct 30 19:18:49.955721 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:49.955684    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-zdb7l"]
Oct 30 19:18:50.347857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:50.347824    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:51.367908 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.367882    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:18:51.403928 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.403904    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:18:51.407270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.407244    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:18:51.452849 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.452807    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-podinfo\") pod \"6d1c4684-874c-4d3d-aebb-517a892cb376\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") "
Oct 30 19:18:51.453050 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.452875    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-kube-api-access-r5snz\" (UniqueName: \"kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-kube-api-access-r5snz\") pod \"6d1c4684-874c-4d3d-aebb-517a892cb376\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") "
Oct 30 19:18:51.453050 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.453002    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-envoy\") pod \"6d1c4684-874c-4d3d-aebb-517a892cb376\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") "
Oct 30 19:18:51.453229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.453068    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d1c4684-874c-4d3d-aebb-517a892cb376-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1c4684-874c-4d3d-aebb-517a892cb376-istiod-ca-cert\") pod \"6d1c4684-874c-4d3d-aebb-517a892cb376\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") "
Oct 30 19:18:51.453229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.453122    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-data\") pod \"6d1c4684-874c-4d3d-aebb-517a892cb376\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") "
Oct 30 19:18:51.453229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.453224    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-socket\") pod \"6d1c4684-874c-4d3d-aebb-517a892cb376\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") "
Oct 30 19:18:51.453391 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.453279    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-certs\") pod \"6d1c4684-874c-4d3d-aebb-517a892cb376\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") "
Oct 30 19:18:51.453391 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.453343    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-credential-socket\") pod \"6d1c4684-874c-4d3d-aebb-517a892cb376\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") "
Oct 30 19:18:51.453484 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.453388    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-token\" (UniqueName: \"kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-token\") pod \"6d1c4684-874c-4d3d-aebb-517a892cb376\" (UID: \"6d1c4684-874c-4d3d-aebb-517a892cb376\") "
Oct 30 19:18:51.453531 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.453488    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-data" pod "6d1c4684-874c-4d3d-aebb-517a892cb376" (UID: "6d1c4684-874c-4d3d-aebb-517a892cb376"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.453611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.453589    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d1c4684-874c-4d3d-aebb-517a892cb376-istiod-ca-cert" pod "6d1c4684-874c-4d3d-aebb-517a892cb376" (UID: "6d1c4684-874c-4d3d-aebb-517a892cb376"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:51.453709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.453688    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-certs" pod "6d1c4684-874c-4d3d-aebb-517a892cb376" (UID: "6d1c4684-874c-4d3d-aebb-517a892cb376"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.453879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.453860    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-socket" pod "6d1c4684-874c-4d3d-aebb-517a892cb376" (UID: "6d1c4684-874c-4d3d-aebb-517a892cb376"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.454051 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.454025    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-credential-socket" pod "6d1c4684-874c-4d3d-aebb-517a892cb376" (UID: "6d1c4684-874c-4d3d-aebb-517a892cb376"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.454206 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.454131    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1c4684-874c-4d3d-aebb-517a892cb376-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.454206 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.454184    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.454206 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.454199    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.454206 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.454207    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.455461 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.455426    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-kube-api-access-r5snz" pod "6d1c4684-874c-4d3d-aebb-517a892cb376" (UID: "6d1c4684-874c-4d3d-aebb-517a892cb376"). InnerVolumeSpecName "kube-api-access-r5snz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:51.455793 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.455776    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-envoy" pod "6d1c4684-874c-4d3d-aebb-517a892cb376" (UID: "6d1c4684-874c-4d3d-aebb-517a892cb376"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.455856 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.455809    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-podinfo" pod "6d1c4684-874c-4d3d-aebb-517a892cb376" (UID: "6d1c4684-874c-4d3d-aebb-517a892cb376"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:18:51.455922 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.455897    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-token" pod "6d1c4684-874c-4d3d-aebb-517a892cb376" (UID: "6d1c4684-874c-4d3d-aebb-517a892cb376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:51.478280 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.478248    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" path="/var/lib/kubelet/pods/a13d3d6b-d787-4b0e-a46a-4a132174f16f/volumes"
Oct 30 19:18:51.478819 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.478805    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d328b8d4-3583-4984-9435-1cb4349ae6aa" path="/var/lib/kubelet/pods/d328b8d4-3583-4984-9435-1cb4349ae6aa/volumes"
Oct 30 19:18:51.555361 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555279    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-socket\") pod \"da67831f-c12e-47db-ae2b-6a9276287b74\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") "
Oct 30 19:18:51.555361 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555332    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da67831f-c12e-47db-ae2b-6a9276287b74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da67831f-c12e-47db-ae2b-6a9276287b74-istio-podinfo\") pod \"da67831f-c12e-47db-ae2b-6a9276287b74\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") "
Oct 30 19:18:51.555609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555373    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-kube-api-access-pmb6w\" (UniqueName: \"kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-kube-api-access-pmb6w\") pod \"da67831f-c12e-47db-ae2b-6a9276287b74\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") "
Oct 30 19:18:51.555609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555407    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-podinfo\") pod \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") "
Oct 30 19:18:51.555609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555433    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-data\") pod \"da67831f-c12e-47db-ae2b-6a9276287b74\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") "
Oct 30 19:18:51.555609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555466    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-envoy\") pod \"da67831f-c12e-47db-ae2b-6a9276287b74\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") "
Oct 30 19:18:51.555609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555497    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da67831f-c12e-47db-ae2b-6a9276287b74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da67831f-c12e-47db-ae2b-6a9276287b74-istiod-ca-cert\") pod \"da67831f-c12e-47db-ae2b-6a9276287b74\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") "
Oct 30 19:18:51.555609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555542    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istiod-ca-cert\") pod \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") "
Oct 30 19:18:51.555924 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555607    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-socket" pod "da67831f-c12e-47db-ae2b-6a9276287b74" (UID: "da67831f-c12e-47db-ae2b-6a9276287b74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.555924 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555632    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-certs\") pod \"da67831f-c12e-47db-ae2b-6a9276287b74\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") "
Oct 30 19:18:51.555924 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555674    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-socket\") pod \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") "
Oct 30 19:18:51.555924 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555692    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-certs\") pod \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") "
Oct 30 19:18:51.555924 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555737    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-token\" (UniqueName: \"kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-token\") pod \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") "
Oct 30 19:18:51.555924 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.555771    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-credential-socket\") pod \"da67831f-c12e-47db-ae2b-6a9276287b74\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") "
Oct 30 19:18:51.556446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.556089    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-certs" pod "da67831f-c12e-47db-ae2b-6a9276287b74" (UID: "da67831f-c12e-47db-ae2b-6a9276287b74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.556446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.556199    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da67831f-c12e-47db-ae2b-6a9276287b74-istiod-ca-cert" pod "da67831f-c12e-47db-ae2b-6a9276287b74" (UID: "da67831f-c12e-47db-ae2b-6a9276287b74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:51.556670 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.556644    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istiod-ca-cert" pod "f84ead0c-c769-4f58-bc0f-e5d7528ae907" (UID: "f84ead0c-c769-4f58-bc0f-e5d7528ae907"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:51.556829 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.556804    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-data" pod "da67831f-c12e-47db-ae2b-6a9276287b74" (UID: "da67831f-c12e-47db-ae2b-6a9276287b74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.557051 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.557028    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-certs" pod "f84ead0c-c769-4f58-bc0f-e5d7528ae907" (UID: "f84ead0c-c769-4f58-bc0f-e5d7528ae907"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.557125 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.557097    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-socket" pod "f84ead0c-c769-4f58-bc0f-e5d7528ae907" (UID: "f84ead0c-c769-4f58-bc0f-e5d7528ae907"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.557342 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.557320    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-credential-socket" pod "da67831f-c12e-47db-ae2b-6a9276287b74" (UID: "da67831f-c12e-47db-ae2b-6a9276287b74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.557533 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.557513    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-istio-token\" (UniqueName: \"kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-istio-token\") pod \"da67831f-c12e-47db-ae2b-6a9276287b74\" (UID: \"da67831f-c12e-47db-ae2b-6a9276287b74\") "
Oct 30 19:18:51.557607 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.557555    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-credential-socket\") pod \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") "
Oct 30 19:18:51.557607 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.557595    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-kube-api-access-2ssnk\" (UniqueName: \"kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-kube-api-access-2ssnk\") pod \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") "
Oct 30 19:18:51.557730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.557628    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-envoy\") pod \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") "
Oct 30 19:18:51.557730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.557685    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-data\") pod \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\" (UID: \"f84ead0c-c769-4f58-bc0f-e5d7528ae907\") "
Oct 30 19:18:51.558374 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558352    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.558476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558378    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.558476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558391    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.558476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558403    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.558476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558420    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.558476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558434    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5snz\" (UniqueName: \"kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-kube-api-access-r5snz\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.558476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558447    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.558476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558459    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.558476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558472    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.559128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558483    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da67831f-c12e-47db-ae2b-6a9276287b74-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.559128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558495    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.559128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558508    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1c4684-874c-4d3d-aebb-517a892cb376-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.559128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558521    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1c4684-874c-4d3d-aebb-517a892cb376-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.559128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558642    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-podinfo" pod "f84ead0c-c769-4f58-bc0f-e5d7528ae907" (UID: "f84ead0c-c769-4f58-bc0f-e5d7528ae907"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:18:51.559128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558719    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-data" pod "f84ead0c-c769-4f58-bc0f-e5d7528ae907" (UID: "f84ead0c-c769-4f58-bc0f-e5d7528ae907"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.559128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558829    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-envoy" pod "da67831f-c12e-47db-ae2b-6a9276287b74" (UID: "da67831f-c12e-47db-ae2b-6a9276287b74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.559128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.558923    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-credential-socket" pod "f84ead0c-c769-4f58-bc0f-e5d7528ae907" (UID: "f84ead0c-c769-4f58-bc0f-e5d7528ae907"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.559596 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.559515    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-token" pod "f84ead0c-c769-4f58-bc0f-e5d7528ae907" (UID: "f84ead0c-c769-4f58-bc0f-e5d7528ae907"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:51.559761 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.559726    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da67831f-c12e-47db-ae2b-6a9276287b74-istio-podinfo" pod "da67831f-c12e-47db-ae2b-6a9276287b74" (UID: "da67831f-c12e-47db-ae2b-6a9276287b74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:18:51.560379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.560357    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-kube-api-access-2ssnk" pod "f84ead0c-c769-4f58-bc0f-e5d7528ae907" (UID: "f84ead0c-c769-4f58-bc0f-e5d7528ae907"). InnerVolumeSpecName "kube-api-access-2ssnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:51.560680 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.560659    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-istio-token" pod "da67831f-c12e-47db-ae2b-6a9276287b74" (UID: "da67831f-c12e-47db-ae2b-6a9276287b74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:51.560729 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.560708    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-envoy" pod "f84ead0c-c769-4f58-bc0f-e5d7528ae907" (UID: "f84ead0c-c769-4f58-bc0f-e5d7528ae907"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.560807 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.560791    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-kube-api-access-pmb6w" pod "da67831f-c12e-47db-ae2b-6a9276287b74" (UID: "da67831f-c12e-47db-ae2b-6a9276287b74"). InnerVolumeSpecName "kube-api-access-pmb6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:51.581518 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.581474    6014 generic.go:356] "Generic (PLEG): container finished" podID="da67831f-c12e-47db-ae2b-6a9276287b74" containerID="739b0abed625d038576d8792f4e8152f235b9dfa0a015f92e7fba509bc3a5346" exitCode=0
Oct 30 19:18:51.581731 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.581707    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm" event={"ID":"da67831f-c12e-47db-ae2b-6a9276287b74","Type":"ContainerDied","Data":"739b0abed625d038576d8792f4e8152f235b9dfa0a015f92e7fba509bc3a5346"}
Oct 30 19:18:51.581822 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.581806    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm" event={"ID":"da67831f-c12e-47db-ae2b-6a9276287b74","Type":"ContainerDied","Data":"85ace61612ab1cc555e512bec37b09508b43c6aefcae309a0e8c9a4c5b44aaa3"}
Oct 30 19:18:51.581876 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.581848    6014 scope.go:122] "RemoveContainer" containerID="739b0abed625d038576d8792f4e8152f235b9dfa0a015f92e7fba509bc3a5346"
Oct 30 19:18:51.582065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.582048    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"
Oct 30 19:18:51.584649 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.584622    6014 generic.go:356] "Generic (PLEG): container finished" podID="f84ead0c-c769-4f58-bc0f-e5d7528ae907" containerID="345454db6c78678434492d6cbdd97ce4008e8bfb856c85b5118776a3662d2f53" exitCode=0
Oct 30 19:18:51.584913 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.584894    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t" event={"ID":"f84ead0c-c769-4f58-bc0f-e5d7528ae907","Type":"ContainerDied","Data":"345454db6c78678434492d6cbdd97ce4008e8bfb856c85b5118776a3662d2f53"}
Oct 30 19:18:51.585043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.585029    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t" event={"ID":"f84ead0c-c769-4f58-bc0f-e5d7528ae907","Type":"ContainerDied","Data":"82c73574dcda053a9036744f7a02b8efbfcba90a7ae6badb89d521700d161888"}
Oct 30 19:18:51.585255 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.585239    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"
Oct 30 19:18:51.588357 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.588328    6014 generic.go:356] "Generic (PLEG): container finished" podID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerID="46283fd697700f9f52b70067fbcda31ca3db86b06fe0ead3384fbcc5b3df205f" exitCode=0
Oct 30 19:18:51.588468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.588364    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" event={"ID":"6d1c4684-874c-4d3d-aebb-517a892cb376","Type":"ContainerDied","Data":"46283fd697700f9f52b70067fbcda31ca3db86b06fe0ead3384fbcc5b3df205f"}
Oct 30 19:18:51.588468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.588416    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"
Oct 30 19:18:51.588577 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.588506    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97" event={"ID":"6d1c4684-874c-4d3d-aebb-517a892cb376","Type":"ContainerDied","Data":"f601df9a9853624aa0c36e65a49c8ae29df1f10d6bd4016f253d46f4e3696ee8"}
Oct 30 19:18:51.605901 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.605872    6014 scope.go:122] "RemoveContainer" containerID="739b0abed625d038576d8792f4e8152f235b9dfa0a015f92e7fba509bc3a5346"
Oct 30 19:18:51.606269 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:51.606248    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739b0abed625d038576d8792f4e8152f235b9dfa0a015f92e7fba509bc3a5346\": container with ID starting with 739b0abed625d038576d8792f4e8152f235b9dfa0a015f92e7fba509bc3a5346 not found: ID does not exist" containerID="739b0abed625d038576d8792f4e8152f235b9dfa0a015f92e7fba509bc3a5346"
Oct 30 19:18:51.606384 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.606276    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739b0abed625d038576d8792f4e8152f235b9dfa0a015f92e7fba509bc3a5346"} err="failed to get container status \"739b0abed625d038576d8792f4e8152f235b9dfa0a015f92e7fba509bc3a5346\": rpc error: code = NotFound desc = could not find container \"739b0abed625d038576d8792f4e8152f235b9dfa0a015f92e7fba509bc3a5346\": container with ID starting with 739b0abed625d038576d8792f4e8152f235b9dfa0a015f92e7fba509bc3a5346 not found: ID does not exist"
Oct 30 19:18:51.606384 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.606294    6014 scope.go:122] "RemoveContainer" containerID="345454db6c78678434492d6cbdd97ce4008e8bfb856c85b5118776a3662d2f53"
Oct 30 19:18:51.616537 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.616512    6014 scope.go:122] "RemoveContainer" containerID="345454db6c78678434492d6cbdd97ce4008e8bfb856c85b5118776a3662d2f53"
Oct 30 19:18:51.616893 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:51.616864    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345454db6c78678434492d6cbdd97ce4008e8bfb856c85b5118776a3662d2f53\": container with ID starting with 345454db6c78678434492d6cbdd97ce4008e8bfb856c85b5118776a3662d2f53 not found: ID does not exist" containerID="345454db6c78678434492d6cbdd97ce4008e8bfb856c85b5118776a3662d2f53"
Oct 30 19:18:51.616972 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.616906    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345454db6c78678434492d6cbdd97ce4008e8bfb856c85b5118776a3662d2f53"} err="failed to get container status \"345454db6c78678434492d6cbdd97ce4008e8bfb856c85b5118776a3662d2f53\": rpc error: code = NotFound desc = could not find container \"345454db6c78678434492d6cbdd97ce4008e8bfb856c85b5118776a3662d2f53\": container with ID starting with 345454db6c78678434492d6cbdd97ce4008e8bfb856c85b5118776a3662d2f53 not found: ID does not exist"
Oct 30 19:18:51.616972 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.616932    6014 scope.go:122] "RemoveContainer" containerID="46283fd697700f9f52b70067fbcda31ca3db86b06fe0ead3384fbcc5b3df205f"
Oct 30 19:18:51.632534 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.632508    6014 scope.go:122] "RemoveContainer" containerID="2bbfb018dd3c8ce4fa048f873c3971203c5acac2502e45a81e7772984231cbe2"
Oct 30 19:18:51.642430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.642408    6014 scope.go:122] "RemoveContainer" containerID="46283fd697700f9f52b70067fbcda31ca3db86b06fe0ead3384fbcc5b3df205f"
Oct 30 19:18:51.642759 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:51.642734    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46283fd697700f9f52b70067fbcda31ca3db86b06fe0ead3384fbcc5b3df205f\": container with ID starting with 46283fd697700f9f52b70067fbcda31ca3db86b06fe0ead3384fbcc5b3df205f not found: ID does not exist" containerID="46283fd697700f9f52b70067fbcda31ca3db86b06fe0ead3384fbcc5b3df205f"
Oct 30 19:18:51.642894 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.642769    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46283fd697700f9f52b70067fbcda31ca3db86b06fe0ead3384fbcc5b3df205f"} err="failed to get container status \"46283fd697700f9f52b70067fbcda31ca3db86b06fe0ead3384fbcc5b3df205f\": rpc error: code = NotFound desc = could not find container \"46283fd697700f9f52b70067fbcda31ca3db86b06fe0ead3384fbcc5b3df205f\": container with ID starting with 46283fd697700f9f52b70067fbcda31ca3db86b06fe0ead3384fbcc5b3df205f not found: ID does not exist"
Oct 30 19:18:51.642894 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.642788    6014 scope.go:122] "RemoveContainer" containerID="2bbfb018dd3c8ce4fa048f873c3971203c5acac2502e45a81e7772984231cbe2"
Oct 30 19:18:51.643094 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:51.643071    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bbfb018dd3c8ce4fa048f873c3971203c5acac2502e45a81e7772984231cbe2\": container with ID starting with 2bbfb018dd3c8ce4fa048f873c3971203c5acac2502e45a81e7772984231cbe2 not found: ID does not exist" containerID="2bbfb018dd3c8ce4fa048f873c3971203c5acac2502e45a81e7772984231cbe2"
Oct 30 19:18:51.643203 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.643101    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbfb018dd3c8ce4fa048f873c3971203c5acac2502e45a81e7772984231cbe2"} err="failed to get container status \"2bbfb018dd3c8ce4fa048f873c3971203c5acac2502e45a81e7772984231cbe2\": rpc error: code = NotFound desc = could not find container \"2bbfb018dd3c8ce4fa048f873c3971203c5acac2502e45a81e7772984231cbe2\": container with ID starting with 2bbfb018dd3c8ce4fa048f873c3971203c5acac2502e45a81e7772984231cbe2 not found: ID does not exist"
Oct 30 19:18:51.659755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.659598    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.659755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.659634    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.659755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.659648    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.659755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.659660    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ssnk\" (UniqueName: \"kubernetes.io/projected/f84ead0c-c769-4f58-bc0f-e5d7528ae907-kube-api-access-2ssnk\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.659755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.659673    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.659755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.659684    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.659755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.659695    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da67831f-c12e-47db-ae2b-6a9276287b74-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.659755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.659707    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmb6w\" (UniqueName: \"kubernetes.io/projected/da67831f-c12e-47db-ae2b-6a9276287b74-kube-api-access-pmb6w\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.659755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.659719    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84ead0c-c769-4f58-bc0f-e5d7528ae907-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.659755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.659731    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da67831f-c12e-47db-ae2b-6a9276287b74-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.937396 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.937318    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"]
Oct 30 19:18:51.940715 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:51.940687    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-m579t"]
Oct 30 19:18:52.049174 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:52.049122    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"]
Oct 30 19:18:52.055829 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:52.055799    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vvksm"]
Oct 30 19:18:52.163736 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:52.163700    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"]
Oct 30 19:18:52.170557 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:52.170529    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-vld97"]
Oct 30 19:18:53.271723 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:18:53.271687    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:18:53.475997 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:53.475965    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" path="/var/lib/kubelet/pods/6d1c4684-874c-4d3d-aebb-517a892cb376/volumes"
Oct 30 19:18:53.476509 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:53.476495    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da67831f-c12e-47db-ae2b-6a9276287b74" path="/var/lib/kubelet/pods/da67831f-c12e-47db-ae2b-6a9276287b74/volumes"
Oct 30 19:18:53.476902 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:53.476890    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84ead0c-c769-4f58-bc0f-e5d7528ae907" path="/var/lib/kubelet/pods/f84ead0c-c769-4f58-bc0f-e5d7528ae907/volumes"
Oct 30 19:18:58.348209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:58.348175    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:58.361519 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:58.361489    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:58.647654 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:58.647569    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:18:59.281208 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:59.281175    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"]
Oct 30 19:18:59.283754 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:59.283733    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-975cf5-4w85g"]
Oct 30 19:18:59.476171 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:18:59.476113    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="152f91b3-d4d9-4cf2-a821-12062ad0fb9b" path="/var/lib/kubelet/pods/152f91b3-d4d9-4cf2-a821-12062ad0fb9b/volumes"
Oct 30 19:19:06.056954 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:06.050615    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"]
Oct 30 19:19:06.056954 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:06.051236    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy" containerID="cri-o://84452e74be4caaf71b7a03af6561f7ba06e30d6fa7fbf9c1b88d13394a33d413" gracePeriod=30
Oct 30 19:19:07.257235 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:07.257187    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"]
Oct 30 19:19:07.257714 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:07.257457    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" podUID="1a7e57df-ec86-4e01-adde-734560a4348e" containerName="istio-proxy" containerID="cri-o://e958355e4a77859035cf69eb1a4a5f52a443f998d69d998a0100aa13c5a02aef" gracePeriod=30
Oct 30 19:19:08.764535 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:08.764502    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"]
Oct 30 19:19:11.330048 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.329981    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 19:19:11.439785 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.439739    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-podinfo\") pod \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") "
Oct 30 19:19:11.439968 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.439835    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-socket\") pod \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") "
Oct 30 19:19:11.439968 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.439874    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-credential-socket\") pod \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") "
Oct 30 19:19:11.439968 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.439930    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-data\") pod \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") "
Oct 30 19:19:11.439968 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.439956    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istiod-ca-cert\") pod \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") "
Oct 30 19:19:11.440224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440012    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-envoy\") pod \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") "
Oct 30 19:19:11.440224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440077    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-certs\") pod \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") "
Oct 30 19:19:11.440224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440111    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-token\" (UniqueName: \"kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-token\") pod \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") "
Oct 30 19:19:11.440224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440197    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-kube-api-access-s99v8\" (UniqueName: \"kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-kube-api-access-s99v8\") pod \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\" (UID: \"cab5db1f-c822-4fe8-a692-6e8aaa8306bb\") "
Oct 30 19:19:11.440224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440203    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-socket" pod "cab5db1f-c822-4fe8-a692-6e8aaa8306bb" (UID: "cab5db1f-c822-4fe8-a692-6e8aaa8306bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:11.440491 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440309    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-credential-socket" pod "cab5db1f-c822-4fe8-a692-6e8aaa8306bb" (UID: "cab5db1f-c822-4fe8-a692-6e8aaa8306bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:11.440491 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440377    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-data" pod "cab5db1f-c822-4fe8-a692-6e8aaa8306bb" (UID: "cab5db1f-c822-4fe8-a692-6e8aaa8306bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:11.440491 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440481    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istiod-ca-cert" pod "cab5db1f-c822-4fe8-a692-6e8aaa8306bb" (UID: "cab5db1f-c822-4fe8-a692-6e8aaa8306bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:11.440620 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440487    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-certs" pod "cab5db1f-c822-4fe8-a692-6e8aaa8306bb" (UID: "cab5db1f-c822-4fe8-a692-6e8aaa8306bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:11.440786 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440764    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.440903 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440790    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.440903 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440801    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.440903 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440810    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.440903 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.440818    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.442275 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.442246    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-podinfo" pod "cab5db1f-c822-4fe8-a692-6e8aaa8306bb" (UID: "cab5db1f-c822-4fe8-a692-6e8aaa8306bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:19:11.442617 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.442600    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-kube-api-access-s99v8" pod "cab5db1f-c822-4fe8-a692-6e8aaa8306bb" (UID: "cab5db1f-c822-4fe8-a692-6e8aaa8306bb"). InnerVolumeSpecName "kube-api-access-s99v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:11.442997 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.442972    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-envoy" pod "cab5db1f-c822-4fe8-a692-6e8aaa8306bb" (UID: "cab5db1f-c822-4fe8-a692-6e8aaa8306bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:11.443095 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.442996    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-token" pod "cab5db1f-c822-4fe8-a692-6e8aaa8306bb" (UID: "cab5db1f-c822-4fe8-a692-6e8aaa8306bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:11.542295 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.542260    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.542295 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.542292    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.542510 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.542305    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s99v8\" (UniqueName: \"kubernetes.io/projected/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-kube-api-access-s99v8\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.542510 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.542321    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab5db1f-c822-4fe8-a692-6e8aaa8306bb-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.786015 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.785921    6014 generic.go:356] "Generic (PLEG): container finished" podID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerID="84452e74be4caaf71b7a03af6561f7ba06e30d6fa7fbf9c1b88d13394a33d413" exitCode=0
Oct 30 19:19:11.786015 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.785958    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"
Oct 30 19:19:11.786015 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.786000    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" event={"ID":"cab5db1f-c822-4fe8-a692-6e8aaa8306bb","Type":"ContainerDied","Data":"84452e74be4caaf71b7a03af6561f7ba06e30d6fa7fbf9c1b88d13394a33d413"}
Oct 30 19:19:11.786307 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.786057    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5" event={"ID":"cab5db1f-c822-4fe8-a692-6e8aaa8306bb","Type":"ContainerDied","Data":"72e753ccc1255e1214824aa4622a2ddf4076607b18e487912608c481b5268dcb"}
Oct 30 19:19:11.786307 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.786085    6014 scope.go:122] "RemoveContainer" containerID="84452e74be4caaf71b7a03af6561f7ba06e30d6fa7fbf9c1b88d13394a33d413"
Oct 30 19:19:11.795899 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.795877    6014 scope.go:122] "RemoveContainer" containerID="01c0757f95ab0889c11dbb6afdaef8757d59c628a0b07e3c719bad68138d0dba"
Oct 30 19:19:11.805653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.805626    6014 scope.go:122] "RemoveContainer" containerID="84452e74be4caaf71b7a03af6561f7ba06e30d6fa7fbf9c1b88d13394a33d413"
Oct 30 19:19:11.805996 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:19:11.805975    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84452e74be4caaf71b7a03af6561f7ba06e30d6fa7fbf9c1b88d13394a33d413\": container with ID starting with 84452e74be4caaf71b7a03af6561f7ba06e30d6fa7fbf9c1b88d13394a33d413 not found: ID does not exist" containerID="84452e74be4caaf71b7a03af6561f7ba06e30d6fa7fbf9c1b88d13394a33d413"
Oct 30 19:19:11.806086 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.806005    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84452e74be4caaf71b7a03af6561f7ba06e30d6fa7fbf9c1b88d13394a33d413"} err="failed to get container status \"84452e74be4caaf71b7a03af6561f7ba06e30d6fa7fbf9c1b88d13394a33d413\": rpc error: code = NotFound desc = could not find container \"84452e74be4caaf71b7a03af6561f7ba06e30d6fa7fbf9c1b88d13394a33d413\": container with ID starting with 84452e74be4caaf71b7a03af6561f7ba06e30d6fa7fbf9c1b88d13394a33d413 not found: ID does not exist"
Oct 30 19:19:11.806086 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.806067    6014 scope.go:122] "RemoveContainer" containerID="01c0757f95ab0889c11dbb6afdaef8757d59c628a0b07e3c719bad68138d0dba"
Oct 30 19:19:11.806393 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:19:11.806376    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c0757f95ab0889c11dbb6afdaef8757d59c628a0b07e3c719bad68138d0dba\": container with ID starting with 01c0757f95ab0889c11dbb6afdaef8757d59c628a0b07e3c719bad68138d0dba not found: ID does not exist" containerID="01c0757f95ab0889c11dbb6afdaef8757d59c628a0b07e3c719bad68138d0dba"
Oct 30 19:19:11.806464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.806401    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c0757f95ab0889c11dbb6afdaef8757d59c628a0b07e3c719bad68138d0dba"} err="failed to get container status \"01c0757f95ab0889c11dbb6afdaef8757d59c628a0b07e3c719bad68138d0dba\": rpc error: code = NotFound desc = could not find container \"01c0757f95ab0889c11dbb6afdaef8757d59c628a0b07e3c719bad68138d0dba\": container with ID starting with 01c0757f95ab0889c11dbb6afdaef8757d59c628a0b07e3c719bad68138d0dba not found: ID does not exist"
Oct 30 19:19:11.948570 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.948529    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"]
Oct 30 19:19:11.951338 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:11.951310    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-lpfj5"]
Oct 30 19:19:12.062478 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.062394    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fl4t5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:19:12.062478 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.062462    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" podUID="1a7e57df-ec86-4e01-adde-734560a4348e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:19:12.536126 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.536093    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:19:12.655207 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.655109    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-kube-api-access-l6zb7\" (UniqueName: \"kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-kube-api-access-l6zb7\") pod \"1a7e57df-ec86-4e01-adde-734560a4348e\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") "
Oct 30 19:19:12.655207 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.655174    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-istio-token\" (UniqueName: \"kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-istio-token\") pod \"1a7e57df-ec86-4e01-adde-734560a4348e\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") "
Oct 30 19:19:12.655454 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.655220    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-data\") pod \"1a7e57df-ec86-4e01-adde-734560a4348e\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") "
Oct 30 19:19:12.655454 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.655363    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-envoy\") pod \"1a7e57df-ec86-4e01-adde-734560a4348e\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") "
Oct 30 19:19:12.655454 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.655402    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-credential-socket\") pod \"1a7e57df-ec86-4e01-adde-734560a4348e\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") "
Oct 30 19:19:12.655620 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.655460    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-socket\") pod \"1a7e57df-ec86-4e01-adde-734560a4348e\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") "
Oct 30 19:19:12.655620 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.655517    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a7e57df-ec86-4e01-adde-734560a4348e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7e57df-ec86-4e01-adde-734560a4348e-istio-podinfo\") pod \"1a7e57df-ec86-4e01-adde-734560a4348e\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") "
Oct 30 19:19:12.655620 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.655571    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7e57df-ec86-4e01-adde-734560a4348e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7e57df-ec86-4e01-adde-734560a4348e-istiod-ca-cert\") pod \"1a7e57df-ec86-4e01-adde-734560a4348e\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") "
Oct 30 19:19:12.655774 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.655620    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-certs\") pod \"1a7e57df-ec86-4e01-adde-734560a4348e\" (UID: \"1a7e57df-ec86-4e01-adde-734560a4348e\") "
Oct 30 19:19:12.655829 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.655777    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-data" pod "1a7e57df-ec86-4e01-adde-734560a4348e" (UID: "1a7e57df-ec86-4e01-adde-734560a4348e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:12.655829 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.655790    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-socket" pod "1a7e57df-ec86-4e01-adde-734560a4348e" (UID: "1a7e57df-ec86-4e01-adde-734560a4348e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:12.656288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.656005    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-certs" pod "1a7e57df-ec86-4e01-adde-734560a4348e" (UID: "1a7e57df-ec86-4e01-adde-734560a4348e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:12.656288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.656198    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.656288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.656218    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.656288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.656231    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.656288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.656270    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-credential-socket" pod "1a7e57df-ec86-4e01-adde-734560a4348e" (UID: "1a7e57df-ec86-4e01-adde-734560a4348e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:12.656602 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.656323    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7e57df-ec86-4e01-adde-734560a4348e-istiod-ca-cert" pod "1a7e57df-ec86-4e01-adde-734560a4348e" (UID: "1a7e57df-ec86-4e01-adde-734560a4348e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:12.658320 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.658291    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-kube-api-access-l6zb7" pod "1a7e57df-ec86-4e01-adde-734560a4348e" (UID: "1a7e57df-ec86-4e01-adde-734560a4348e"). InnerVolumeSpecName "kube-api-access-l6zb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:12.658607 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.658587    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a7e57df-ec86-4e01-adde-734560a4348e-istio-podinfo" pod "1a7e57df-ec86-4e01-adde-734560a4348e" (UID: "1a7e57df-ec86-4e01-adde-734560a4348e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:19:12.658813 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.658785    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-envoy" pod "1a7e57df-ec86-4e01-adde-734560a4348e" (UID: "1a7e57df-ec86-4e01-adde-734560a4348e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:12.659329 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.659309    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-istio-token" pod "1a7e57df-ec86-4e01-adde-734560a4348e" (UID: "1a7e57df-ec86-4e01-adde-734560a4348e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:12.713161 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.713104    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:19:12.757521 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.757484    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.757521 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.757512    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.757521 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.757526    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e57df-ec86-4e01-adde-734560a4348e-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.757793 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.757539    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7e57df-ec86-4e01-adde-734560a4348e-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.757793 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.757551    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7e57df-ec86-4e01-adde-734560a4348e-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.757793 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.757563    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6zb7\" (UniqueName: \"kubernetes.io/projected/1a7e57df-ec86-4e01-adde-734560a4348e-kube-api-access-l6zb7\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.794828 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.794796    6014 generic.go:356] "Generic (PLEG): container finished" podID="1a7e57df-ec86-4e01-adde-734560a4348e" containerID="e958355e4a77859035cf69eb1a4a5f52a443f998d69d998a0100aa13c5a02aef" exitCode=0
Oct 30 19:19:12.795009 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.794893    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" event={"ID":"1a7e57df-ec86-4e01-adde-734560a4348e","Type":"ContainerDied","Data":"e958355e4a77859035cf69eb1a4a5f52a443f998d69d998a0100aa13c5a02aef"}
Oct 30 19:19:12.795009 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.794908    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"
Oct 30 19:19:12.795009 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.794952    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5" event={"ID":"1a7e57df-ec86-4e01-adde-734560a4348e","Type":"ContainerDied","Data":"b0b357f0ac9782e1fbb8e47e43c444245e40d3810c0a0838c1c113b744c34731"}
Oct 30 19:19:12.795009 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.794975    6014 scope.go:122] "RemoveContainer" containerID="e958355e4a77859035cf69eb1a4a5f52a443f998d69d998a0100aa13c5a02aef"
Oct 30 19:19:12.806546 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.806527    6014 scope.go:122] "RemoveContainer" containerID="e958355e4a77859035cf69eb1a4a5f52a443f998d69d998a0100aa13c5a02aef"
Oct 30 19:19:12.806824 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:19:12.806806    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e958355e4a77859035cf69eb1a4a5f52a443f998d69d998a0100aa13c5a02aef\": container with ID starting with e958355e4a77859035cf69eb1a4a5f52a443f998d69d998a0100aa13c5a02aef not found: ID does not exist" containerID="e958355e4a77859035cf69eb1a4a5f52a443f998d69d998a0100aa13c5a02aef"
Oct 30 19:19:12.806884 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.806837    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e958355e4a77859035cf69eb1a4a5f52a443f998d69d998a0100aa13c5a02aef"} err="failed to get container status \"e958355e4a77859035cf69eb1a4a5f52a443f998d69d998a0100aa13c5a02aef\": rpc error: code = NotFound desc = could not find container \"e958355e4a77859035cf69eb1a4a5f52a443f998d69d998a0100aa13c5a02aef\": container with ID starting with e958355e4a77859035cf69eb1a4a5f52a443f998d69d998a0100aa13c5a02aef not found: ID does not exist"
Oct 30 19:19:12.957306 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.957219    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"]
Oct 30 19:19:12.961892 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:12.961865    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fl4t5"]
Oct 30 19:19:13.476842 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:13.476810    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7e57df-ec86-4e01-adde-734560a4348e" path="/var/lib/kubelet/pods/1a7e57df-ec86-4e01-adde-734560a4348e/volumes"
Oct 30 19:19:13.477314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:13.477301    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" path="/var/lib/kubelet/pods/cab5db1f-c822-4fe8-a692-6e8aaa8306bb/volumes"
Oct 30 19:19:15.631657 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:15.631613    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"]
Oct 30 19:19:16.725951 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:16.725917    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"]
Oct 30 19:19:16.730256 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:16.730226    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-36d676-xnd57"]
Oct 30 19:19:17.476490 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:17.476448    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8897a808-dd04-489e-96f6-f5506ebe5d5a" path="/var/lib/kubelet/pods/8897a808-dd04-489e-96f6-f5506ebe5d5a/volumes"
Oct 30 19:19:19.521152 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521103    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"]
Oct 30 19:19:19.521673 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521660    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1a7e57df-ec86-4e01-adde-734560a4348e" containerName="istio-proxy"
Oct 30 19:19:19.521719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521674    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7e57df-ec86-4e01-adde-734560a4348e" containerName="istio-proxy"
Oct 30 19:19:19.521719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521686    6014 cpu_manager.go:414] "containerMap: removing container" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerName="istio-proxy"
Oct 30 19:19:19.521719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521693    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerName="istio-proxy"
Oct 30 19:19:19.521719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521702    6014 cpu_manager.go:414] "containerMap: removing container" podUID="d328b8d4-3583-4984-9435-1cb4349ae6aa" containerName="istio-proxy"
Oct 30 19:19:19.521719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521707    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d328b8d4-3583-4984-9435-1cb4349ae6aa" containerName="istio-proxy"
Oct 30 19:19:19.521879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521722    6014 cpu_manager.go:414] "containerMap: removing container" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerName="istio-proxy"
Oct 30 19:19:19.521879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521728    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerName="istio-proxy"
Oct 30 19:19:19.521879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521734    6014 cpu_manager.go:414] "containerMap: removing container" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy"
Oct 30 19:19:19.521879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521739    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy"
Oct 30 19:19:19.521879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521748    6014 cpu_manager.go:414] "containerMap: removing container" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerName="istio-proxy"
Oct 30 19:19:19.521879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521752    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerName="istio-proxy"
Oct 30 19:19:19.521879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521763    6014 cpu_manager.go:414] "containerMap: removing container" podUID="da67831f-c12e-47db-ae2b-6a9276287b74" containerName="istio-proxy"
Oct 30 19:19:19.521879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521767    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da67831f-c12e-47db-ae2b-6a9276287b74" containerName="istio-proxy"
Oct 30 19:19:19.521879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521776    6014 cpu_manager.go:414] "containerMap: removing container" podUID="f84ead0c-c769-4f58-bc0f-e5d7528ae907" containerName="istio-proxy"
Oct 30 19:19:19.521879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521780    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84ead0c-c769-4f58-bc0f-e5d7528ae907" containerName="istio-proxy"
Oct 30 19:19:19.521879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521798    6014 cpu_manager.go:414] "containerMap: removing container" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy"
Oct 30 19:19:19.521879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521802    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy"
Oct 30 19:19:19.522258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521884    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84ead0c-c769-4f58-bc0f-e5d7528ae907" containerName="istio-proxy"
Oct 30 19:19:19.522258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521896    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy"
Oct 30 19:19:19.522258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521901    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerName="istio-proxy"
Oct 30 19:19:19.522258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521912    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7e57df-ec86-4e01-adde-734560a4348e" containerName="istio-proxy"
Oct 30 19:19:19.522258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521918    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="d328b8d4-3583-4984-9435-1cb4349ae6aa" containerName="istio-proxy"
Oct 30 19:19:19.522258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521927    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerName="istio-proxy"
Oct 30 19:19:19.522258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521933    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab5db1f-c822-4fe8-a692-6e8aaa8306bb" containerName="istio-proxy"
Oct 30 19:19:19.522258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521941    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13d3d6b-d787-4b0e-a46a-4a132174f16f" containerName="istio-proxy"
Oct 30 19:19:19.522258 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.521961    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="da67831f-c12e-47db-ae2b-6a9276287b74" containerName="istio-proxy"
Oct 30 19:19:19.525612 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.525594    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.628992 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.628958    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-714b2c\"/\"default-dockercfg-624r6\""
Oct 30 19:19:19.629253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.628969    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-714b2c\"/\"test-model-714b2c-kserve-self-signed-certs\""
Oct 30 19:19:19.631430 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.631411    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"]
Oct 30 19:19:19.639116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.639091    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-model-cache\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.639252 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.639123    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-dshm\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.639349 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.639326    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tls-certs\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.639448 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.639435    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tmp-dir\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.639493 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.639455    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt94t\" (UniqueName: \"kubernetes.io/projected/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-kube-api-access-mt94t\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.639528 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.639506    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-home\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.651272 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.651246    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-714b2c\"/\"openshift-service-ca.crt\""
Oct 30 19:19:19.651391 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.651289    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-714b2c\"/\"kube-root-ca.crt\""
Oct 30 19:19:19.741078 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.741043    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-model-cache\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.741078 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.741082    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-dshm\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.741415 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.741302    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tls-certs\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.741618 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.741592    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tmp-dir\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.741618 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.741605    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-model-cache\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.741807 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.741634    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt94t\" (UniqueName: \"kubernetes.io/projected/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-kube-api-access-mt94t\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.741807 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.741702    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-home\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.742029 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.742010    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-home\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.742090 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.742028    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tmp-dir\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.743625 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.743597    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-dshm\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.744024 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.744004    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tls-certs\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.798640 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.798567    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt94t\" (UniqueName: \"kubernetes.io/projected/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-kube-api-access-mt94t\") pod \"test-model-714b2c-kserve-7c578f7469-8nk2q\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") " pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:19.837982 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:19.837952    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:20.214741 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:20.214716    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"]
Oct 30 19:19:20.217221 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:19:20.217191    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1752af1b_aa7b_4c37_8e5d_0ebc54622df0.slice/crio-000f5575ececab8367d0831f8925088463649e3e01e2eb61cf9862b020d8476d WatchSource:0}: Error finding container 000f5575ececab8367d0831f8925088463649e3e01e2eb61cf9862b020d8476d: Status 404 returned error can't find the container with id 000f5575ececab8367d0831f8925088463649e3e01e2eb61cf9862b020d8476d
Oct 30 19:19:20.632131 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:20.632091    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs" podUID="7c1d3787-e640-4027-b267-88a3c8a19873" containerName="payload-pre-processing" containerID="cri-o://1cdc9d72dfa04dd991fb99eb86ef431cacb95d8567f9531c8bc8fe5a4ebfc73b" gracePeriod=25
Oct 30 19:19:20.881410 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:20.881371    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q" event={"ID":"1752af1b-aa7b-4c37-8e5d-0ebc54622df0","Type":"ContainerStarted","Data":"cfe05ceb1996c09d8355166eeb83854932b89beb08c4ddad438194fbf6fc6d6c"}
Oct 30 19:19:20.881593 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:20.881421    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q" event={"ID":"1752af1b-aa7b-4c37-8e5d-0ebc54622df0","Type":"ContainerStarted","Data":"000f5575ececab8367d0831f8925088463649e3e01e2eb61cf9862b020d8476d"}
Oct 30 19:19:20.995680 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:20.995572    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q" podStartSLOduration=1.995558016 podStartE2EDuration="1.995558016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:20.994484358 +0000 UTC m=+2399.637651974" watchObservedRunningTime="2026-10-30 19:19:20.995558016 +0000 UTC m=+2399.638725629"
Oct 30 19:19:21.838489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:21.838453    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:22.340569 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.340535    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"]
Oct 30 19:19:22.340817 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.340777    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g" podUID="6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" containerName="istio-proxy" containerID="cri-o://74e9bcfbbf4cae0c326c2ba13d0b958bade252b6a100722f793c0fd20c22d9e3" gracePeriod=30
Oct 30 19:19:22.489510 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.489458    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"]
Oct 30 19:19:22.490420 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.490393    6014 cpu_manager.go:414] "containerMap: removing container" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerName="istio-proxy"
Oct 30 19:19:22.490574 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.490422    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerName="istio-proxy"
Oct 30 19:19:22.490654 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.490613    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1c4684-874c-4d3d-aebb-517a892cb376" containerName="istio-proxy"
Oct 30 19:19:22.494682 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.494657    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.586910 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.586874    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.587118 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.586966    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.587118 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.586992    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.587118 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.587020    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.587118 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.587115    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.587468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.587190    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c357435d-3e86-4482-8419-a4050103c921-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.589868 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.589841    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"]
Oct 30 19:19:22.688231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.688195    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.688231 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.688240    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.688496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.688268    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.688496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.688301    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq46w\" (UniqueName: \"kubernetes.io/projected/c357435d-3e86-4482-8419-a4050103c921-kube-api-access-vq46w\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.688496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.688354    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c357435d-3e86-4482-8419-a4050103c921-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.688496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.688386    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c357435d-3e86-4482-8419-a4050103c921-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.688643 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.688548    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.688643 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.688610    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c357435d-3e86-4482-8419-a4050103c921-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.688730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.688636    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.688730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.688685    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.688730 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.688699    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.688841 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.688755    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.688900 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.688866    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.691047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.691017    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c357435d-3e86-4482-8419-a4050103c921-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.691267 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.691248    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c357435d-3e86-4482-8419-a4050103c921-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.789719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.789684    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq46w\" (UniqueName: \"kubernetes.io/projected/c357435d-3e86-4482-8419-a4050103c921-kube-api-access-vq46w\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.789901 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.789738    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c357435d-3e86-4482-8419-a4050103c921-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.789901 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.789760    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c357435d-3e86-4482-8419-a4050103c921-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.790414 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.790396    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c357435d-3e86-4482-8419-a4050103c921-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.851790 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.851757    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c357435d-3e86-4482-8419-a4050103c921-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:22.852184 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:22.851761    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq46w\" (UniqueName: \"kubernetes.io/projected/c357435d-3e86-4482-8419-a4050103c921-kube-api-access-vq46w\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5s95d\" (UID: \"c357435d-3e86-4482-8419-a4050103c921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:23.110614 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:23.110579    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:23.260320 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:19:23.260282    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:19:23.543065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:23.543041    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"]
Oct 30 19:19:23.544780 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:19:23.544754    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc357435d_3e86_4482_8419_a4050103c921.slice/crio-674087cd994c22c02c811a84ba7addaabf4b98aa9520013ebdd46f009cf4cb04 WatchSource:0}: Error finding container 674087cd994c22c02c811a84ba7addaabf4b98aa9520013ebdd46f009cf4cb04: Status 404 returned error can't find the container with id 674087cd994c22c02c811a84ba7addaabf4b98aa9520013ebdd46f009cf4cb04
Oct 30 19:19:23.547323 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:23.547292    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:23.547404 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:23.547359    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:23.547404 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:23.547395    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:23.786705 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:23.786657    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main" containerID="cri-o://43f9da463fc74873ce353be3150ac1f7e17d41967e007ca8dd4a2c84fa2d14ea" gracePeriod=45
Oct 30 19:19:23.904338 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:23.904295    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d" event={"ID":"c357435d-3e86-4482-8419-a4050103c921","Type":"ContainerStarted","Data":"cf2def82906ff7afa362ada0b1669ed05a31d896b10091f22ebce26a6a391292"}
Oct 30 19:19:23.905053 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:23.904363    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d" event={"ID":"c357435d-3e86-4482-8419-a4050103c921","Type":"ContainerStarted","Data":"674087cd994c22c02c811a84ba7addaabf4b98aa9520013ebdd46f009cf4cb04"}
Oct 30 19:19:23.907339 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:23.907311    6014 generic.go:356] "Generic (PLEG): container finished" podID="63526e12-eb15-4369-a74c-8b77f983da8d" containerID="43f9da463fc74873ce353be3150ac1f7e17d41967e007ca8dd4a2c84fa2d14ea" exitCode=0
Oct 30 19:19:23.907463 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:23.907362    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" event={"ID":"63526e12-eb15-4369-a74c-8b77f983da8d","Type":"ContainerDied","Data":"43f9da463fc74873ce353be3150ac1f7e17d41967e007ca8dd4a2c84fa2d14ea"}
Oct 30 19:19:24.025417 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.025353    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d" podStartSLOduration=2.025337041 podStartE2EDuration="2.025337041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:24.021640888 +0000 UTC m=+2402.664808506" watchObservedRunningTime="2026-10-30 19:19:24.025337041 +0000 UTC m=+2402.668504654"
Oct 30 19:19:24.110167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.110104    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:19:24.111129 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.111105    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:24.112605 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.112583    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5s95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:19:24.112719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.112628    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d" podUID="c357435d-3e86-4482-8419-a4050103c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 19:19:24.122593 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.122568    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63526e12-eb15-4369-a74c-8b77f983da8d-kube-api-access-tqtq7\" (UniqueName: \"kubernetes.io/projected/63526e12-eb15-4369-a74c-8b77f983da8d-kube-api-access-tqtq7\") pod \"63526e12-eb15-4369-a74c-8b77f983da8d\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") "
Oct 30 19:19:24.122713 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.122614    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63526e12-eb15-4369-a74c-8b77f983da8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/63526e12-eb15-4369-a74c-8b77f983da8d-tls-certs\") pod \"63526e12-eb15-4369-a74c-8b77f983da8d\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") "
Oct 30 19:19:24.123020 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.122989    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-dshm\") pod \"63526e12-eb15-4369-a74c-8b77f983da8d\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") "
Oct 30 19:19:24.123152 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.123047    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-home\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-home\") pod \"63526e12-eb15-4369-a74c-8b77f983da8d\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") "
Oct 30 19:19:24.123152 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.123068    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-tmp-dir\") pod \"63526e12-eb15-4369-a74c-8b77f983da8d\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") "
Oct 30 19:19:24.123152 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.123115    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-model-cache\") pod \"63526e12-eb15-4369-a74c-8b77f983da8d\" (UID: \"63526e12-eb15-4369-a74c-8b77f983da8d\") "
Oct 30 19:19:24.123445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.123417    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-tmp-dir" pod "63526e12-eb15-4369-a74c-8b77f983da8d" (UID: "63526e12-eb15-4369-a74c-8b77f983da8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:24.123445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.123431    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-model-cache" pod "63526e12-eb15-4369-a74c-8b77f983da8d" (UID: "63526e12-eb15-4369-a74c-8b77f983da8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:24.123573 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.123547    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-home" pod "63526e12-eb15-4369-a74c-8b77f983da8d" (UID: "63526e12-eb15-4369-a74c-8b77f983da8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:24.124185 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.124128    6014 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-home\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.124290 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.124190    6014 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-tmp-dir\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.124290 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.124205    6014 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-model-cache\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.125128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.125103    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63526e12-eb15-4369-a74c-8b77f983da8d-tls-certs" pod "63526e12-eb15-4369-a74c-8b77f983da8d" (UID: "63526e12-eb15-4369-a74c-8b77f983da8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:24.125128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.125103    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63526e12-eb15-4369-a74c-8b77f983da8d-kube-api-access-tqtq7" pod "63526e12-eb15-4369-a74c-8b77f983da8d" (UID: "63526e12-eb15-4369-a74c-8b77f983da8d"). InnerVolumeSpecName "kube-api-access-tqtq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:24.125358 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.125333    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-dshm" pod "63526e12-eb15-4369-a74c-8b77f983da8d" (UID: "63526e12-eb15-4369-a74c-8b77f983da8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:24.225362 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.225327    6014 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63526e12-eb15-4369-a74c-8b77f983da8d-dshm\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.225362 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.225358    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqtq7\" (UniqueName: \"kubernetes.io/projected/63526e12-eb15-4369-a74c-8b77f983da8d-kube-api-access-tqtq7\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.225362 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.225367    6014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63526e12-eb15-4369-a74c-8b77f983da8d-tls-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.746340 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.746303    6014 scope.go:122] "RemoveContainer" containerID="43f9da463fc74873ce353be3150ac1f7e17d41967e007ca8dd4a2c84fa2d14ea"
Oct 30 19:19:24.913486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.913450    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447" event={"ID":"63526e12-eb15-4369-a74c-8b77f983da8d","Type":"ContainerDied","Data":"9eb891647ab8b9d5200d9a842773b816254e99c328645483e2b5be0c135ac4d5"}
Oct 30 19:19:24.913486 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:24.913473    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"
Oct 30 19:19:25.111741 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:25.111709    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5s95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:19:25.111924 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:25.111772    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d" podUID="c357435d-3e86-4482-8419-a4050103c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 19:19:25.167194 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:25.166379    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"]
Oct 30 19:19:25.175688 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:25.175636    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-611053/simulator-w5-611053-kserve-6b94f9d85b-82447"]
Oct 30 19:19:25.201575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:25.201537    6014 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:19:25.201724 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:25.201590    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g" podUID="6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:19:25.211827 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:25.211796    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs" podUID="7c1d3787-e640-4027-b267-88a3c8a19873" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:19:25.478921 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:25.478836    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" path="/var/lib/kubelet/pods/63526e12-eb15-4369-a74c-8b77f983da8d/volumes"
Oct 30 19:19:26.111583 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:26.111545    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5s95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:19:26.112007 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:26.111603    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d" podUID="c357435d-3e86-4482-8419-a4050103c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 19:19:27.111782 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.111749    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5s95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:19:27.112226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.111804    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d" podUID="c357435d-3e86-4482-8419-a4050103c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 19:19:27.684284 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.684258    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 19:19:27.790128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790096    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-socket\") pod \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") "
Oct 30 19:19:27.790327 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790205    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-token\" (UniqueName: \"kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-token\") pod \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") "
Oct 30 19:19:27.790327 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790256    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-kube-api-access-rf524\" (UniqueName: \"kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-kube-api-access-rf524\") pod \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") "
Oct 30 19:19:27.790327 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790284    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-data\") pod \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") "
Oct 30 19:19:27.790327 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790303    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-envoy\") pod \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") "
Oct 30 19:19:27.790554 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790349    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istiod-ca-cert\") pod \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") "
Oct 30 19:19:27.790554 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790372    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-certs\") pod \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") "
Oct 30 19:19:27.790554 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790391    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-credential-socket\") pod \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") "
Oct 30 19:19:27.790554 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790410    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-socket" pod "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" (UID: "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:27.790554 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790437    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-podinfo\") pod \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\" (UID: \"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb\") "
Oct 30 19:19:27.790819 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790703    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-certs" pod "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" (UID: "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:27.790819 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790734    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istiod-ca-cert" pod "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" (UID: "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:27.790927 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790836    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-credential-socket" pod "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" (UID: "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:27.790927 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.790910    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-data" pod "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" (UID: "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:27.791266 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.791238    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.791266 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.791268    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.791477 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.791283    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.791477 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.791296    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.791477 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.791312    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.792849 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.792825    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-token" pod "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" (UID: "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:27.792952 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.792863    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-kube-api-access-rf524" pod "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" (UID: "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb"). InnerVolumeSpecName "kube-api-access-rf524". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:27.793611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.793592    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-envoy" pod "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" (UID: "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:27.793668 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.793608    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-podinfo" pod "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" (UID: "6f71e791-7dc8-41b8-aa8c-e3c4b21779cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:19:27.892763 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.892678    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.892763 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.892715    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf524\" (UniqueName: \"kubernetes.io/projected/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-kube-api-access-rf524\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.892763 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.892730    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.892763 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.892743    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.933037 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.933002    6014 generic.go:356] "Generic (PLEG): container finished" podID="6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" containerID="74e9bcfbbf4cae0c326c2ba13d0b958bade252b6a100722f793c0fd20c22d9e3" exitCode=0
Oct 30 19:19:27.933246 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.933069    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"
Oct 30 19:19:27.933246 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.933080    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g" event={"ID":"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb","Type":"ContainerDied","Data":"74e9bcfbbf4cae0c326c2ba13d0b958bade252b6a100722f793c0fd20c22d9e3"}
Oct 30 19:19:27.933246 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.933164    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g" event={"ID":"6f71e791-7dc8-41b8-aa8c-e3c4b21779cb","Type":"ContainerDied","Data":"6a041edf64d41292e4bc0138dd989ee8d088cffa72b9d5ad56af113e3a604560"}
Oct 30 19:19:27.933246 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.933189    6014 scope.go:122] "RemoveContainer" containerID="74e9bcfbbf4cae0c326c2ba13d0b958bade252b6a100722f793c0fd20c22d9e3"
Oct 30 19:19:27.943709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.943688    6014 scope.go:122] "RemoveContainer" containerID="74e9bcfbbf4cae0c326c2ba13d0b958bade252b6a100722f793c0fd20c22d9e3"
Oct 30 19:19:27.943992 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:19:27.943972    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e9bcfbbf4cae0c326c2ba13d0b958bade252b6a100722f793c0fd20c22d9e3\": container with ID starting with 74e9bcfbbf4cae0c326c2ba13d0b958bade252b6a100722f793c0fd20c22d9e3 not found: ID does not exist" containerID="74e9bcfbbf4cae0c326c2ba13d0b958bade252b6a100722f793c0fd20c22d9e3"
Oct 30 19:19:27.944047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:27.944003    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e9bcfbbf4cae0c326c2ba13d0b958bade252b6a100722f793c0fd20c22d9e3"} err="failed to get container status \"74e9bcfbbf4cae0c326c2ba13d0b958bade252b6a100722f793c0fd20c22d9e3\": rpc error: code = NotFound desc = could not find container \"74e9bcfbbf4cae0c326c2ba13d0b958bade252b6a100722f793c0fd20c22d9e3\": container with ID starting with 74e9bcfbbf4cae0c326c2ba13d0b958bade252b6a100722f793c0fd20c22d9e3 not found: ID does not exist"
Oct 30 19:19:28.101293 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:28.101251    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"]
Oct 30 19:19:28.109386 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:28.109350    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-rc62g"]
Oct 30 19:19:28.111920 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:28.111895    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5s95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:19:28.112273 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:28.111952    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d" podUID="c357435d-3e86-4482-8419-a4050103c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 19:19:29.111502 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:29.111461    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5s95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:19:29.111705 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:29.111519    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d" podUID="c357435d-3e86-4482-8419-a4050103c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 19:19:29.479808 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:29.479728    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" path="/var/lib/kubelet/pods/6f71e791-7dc8-41b8-aa8c-e3c4b21779cb/volumes"
Oct 30 19:19:29.838975 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:29.838930    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:29.852366 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:29.852329    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:29.957000 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:29.956973    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:19:30.111928 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:30.111830    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5s95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:19:30.111928 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:30.111893    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d" podUID="c357435d-3e86-4482-8419-a4050103c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 19:19:31.111277 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:31.111235    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5s95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:19:31.111729 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:31.111294    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d" podUID="c357435d-3e86-4482-8419-a4050103c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 19:19:32.115445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:32.115413    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:32.115918 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:32.115849    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:32.116860 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:32.116840    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d"
Oct 30 19:19:35.212276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:35.212242    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs" podUID="7c1d3787-e640-4027-b267-88a3c8a19873" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:19:36.837492 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:36.837452    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"]
Oct 30 19:19:40.911166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:40.911110    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"
Oct 30 19:19:41.019515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.019482    6014 generic.go:356] "Generic (PLEG): container finished" podID="7c1d3787-e640-4027-b267-88a3c8a19873" containerID="1cdc9d72dfa04dd991fb99eb86ef431cacb95d8567f9531c8bc8fe5a4ebfc73b" exitCode=0
Oct 30 19:19:41.019684 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.019525    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs" event={"ID":"7c1d3787-e640-4027-b267-88a3c8a19873","Type":"ContainerDied","Data":"1cdc9d72dfa04dd991fb99eb86ef431cacb95d8567f9531c8bc8fe5a4ebfc73b"}
Oct 30 19:19:41.019684 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.019549    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"
Oct 30 19:19:41.019684 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.019572    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs" event={"ID":"7c1d3787-e640-4027-b267-88a3c8a19873","Type":"ContainerDied","Data":"ecf37a622416e5e45940bc30db8d625f44f2c35721b7f21eeb2e5e2072d2d75a"}
Oct 30 19:19:41.019684 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.019599    6014 scope.go:122] "RemoveContainer" containerID="1cdc9d72dfa04dd991fb99eb86ef431cacb95d8567f9531c8bc8fe5a4ebfc73b"
Oct 30 19:19:41.029041 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.029021    6014 scope.go:122] "RemoveContainer" containerID="1cdc9d72dfa04dd991fb99eb86ef431cacb95d8567f9531c8bc8fe5a4ebfc73b"
Oct 30 19:19:41.029355 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:19:41.029325    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cdc9d72dfa04dd991fb99eb86ef431cacb95d8567f9531c8bc8fe5a4ebfc73b\": container with ID starting with 1cdc9d72dfa04dd991fb99eb86ef431cacb95d8567f9531c8bc8fe5a4ebfc73b not found: ID does not exist" containerID="1cdc9d72dfa04dd991fb99eb86ef431cacb95d8567f9531c8bc8fe5a4ebfc73b"
Oct 30 19:19:41.029455 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.029357    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cdc9d72dfa04dd991fb99eb86ef431cacb95d8567f9531c8bc8fe5a4ebfc73b"} err="failed to get container status \"1cdc9d72dfa04dd991fb99eb86ef431cacb95d8567f9531c8bc8fe5a4ebfc73b\": rpc error: code = NotFound desc = could not find container \"1cdc9d72dfa04dd991fb99eb86ef431cacb95d8567f9531c8bc8fe5a4ebfc73b\": container with ID starting with 1cdc9d72dfa04dd991fb99eb86ef431cacb95d8567f9531c8bc8fe5a4ebfc73b not found: ID does not exist"
Oct 30 19:19:41.053186 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.053133    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1d3787-e640-4027-b267-88a3c8a19873-kube-api-access-xgmxj\" (UniqueName: \"kubernetes.io/projected/7c1d3787-e640-4027-b267-88a3c8a19873-kube-api-access-xgmxj\") pod \"7c1d3787-e640-4027-b267-88a3c8a19873\" (UID: \"7c1d3787-e640-4027-b267-88a3c8a19873\") "
Oct 30 19:19:41.053354 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.053207    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1d3787-e640-4027-b267-88a3c8a19873-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1d3787-e640-4027-b267-88a3c8a19873-plugins-config-volume\") pod \"7c1d3787-e640-4027-b267-88a3c8a19873\" (UID: \"7c1d3787-e640-4027-b267-88a3c8a19873\") "
Oct 30 19:19:41.053647 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.053624    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1d3787-e640-4027-b267-88a3c8a19873-plugins-config-volume" pod "7c1d3787-e640-4027-b267-88a3c8a19873" (UID: "7c1d3787-e640-4027-b267-88a3c8a19873"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:41.053753 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.053736    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1d3787-e640-4027-b267-88a3c8a19873-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:41.055649 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.055617    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1d3787-e640-4027-b267-88a3c8a19873-kube-api-access-xgmxj" pod "7c1d3787-e640-4027-b267-88a3c8a19873" (UID: "7c1d3787-e640-4027-b267-88a3c8a19873"). InnerVolumeSpecName "kube-api-access-xgmxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:41.155182 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.155064    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgmxj\" (UniqueName: \"kubernetes.io/projected/7c1d3787-e640-4027-b267-88a3c8a19873-kube-api-access-xgmxj\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:41.438487 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.438406    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"]
Oct 30 19:19:41.488112 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.488086    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-611053-7848c966bc-cxdhs"]
Oct 30 19:19:41.838251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:41.838214    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4" podUID="1b9a80a2-6c53-40a3-a5a3-44257e705082" containerName="payload-processing" containerID="cri-o://5ecd9f25b19dc016cb76d7f36dff3d8a6314842c1461bf7bd5443714674746fa" gracePeriod=25
Oct 30 19:19:42.030724 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:42.030693    6014 generic.go:356] "Generic (PLEG): container finished" podID="1b9a80a2-6c53-40a3-a5a3-44257e705082" containerID="5ecd9f25b19dc016cb76d7f36dff3d8a6314842c1461bf7bd5443714674746fa" exitCode=0
Oct 30 19:19:42.031293 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:42.030917    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4" event={"ID":"1b9a80a2-6c53-40a3-a5a3-44257e705082","Type":"ContainerDied","Data":"5ecd9f25b19dc016cb76d7f36dff3d8a6314842c1461bf7bd5443714674746fa"}
Oct 30 19:19:42.098073 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:42.097996    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"
Oct 30 19:19:42.179751 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:42.179712    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b9a80a2-6c53-40a3-a5a3-44257e705082-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9a80a2-6c53-40a3-a5a3-44257e705082-plugins-config-volume\") pod \"1b9a80a2-6c53-40a3-a5a3-44257e705082\" (UID: \"1b9a80a2-6c53-40a3-a5a3-44257e705082\") "
Oct 30 19:19:42.179751 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:42.179754    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9a80a2-6c53-40a3-a5a3-44257e705082-kube-api-access-c8x2j\" (UniqueName: \"kubernetes.io/projected/1b9a80a2-6c53-40a3-a5a3-44257e705082-kube-api-access-c8x2j\") pod \"1b9a80a2-6c53-40a3-a5a3-44257e705082\" (UID: \"1b9a80a2-6c53-40a3-a5a3-44257e705082\") "
Oct 30 19:19:42.180124 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:42.180099    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9a80a2-6c53-40a3-a5a3-44257e705082-plugins-config-volume" pod "1b9a80a2-6c53-40a3-a5a3-44257e705082" (UID: "1b9a80a2-6c53-40a3-a5a3-44257e705082"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:42.180281 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:42.180266    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9a80a2-6c53-40a3-a5a3-44257e705082-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:42.182051 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:42.182021    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9a80a2-6c53-40a3-a5a3-44257e705082-kube-api-access-c8x2j" pod "1b9a80a2-6c53-40a3-a5a3-44257e705082" (UID: "1b9a80a2-6c53-40a3-a5a3-44257e705082"). InnerVolumeSpecName "kube-api-access-c8x2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:42.281033 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:42.280997    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8x2j\" (UniqueName: \"kubernetes.io/projected/1b9a80a2-6c53-40a3-a5a3-44257e705082-kube-api-access-c8x2j\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:43.038952 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:43.038914    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4" event={"ID":"1b9a80a2-6c53-40a3-a5a3-44257e705082","Type":"ContainerDied","Data":"341f2ef2354a417b1d20e39e21879ddc6f9f02053e71d1a7782b2611d46faca3"}
Oct 30 19:19:43.039441 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:43.038962    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"
Oct 30 19:19:43.039441 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:43.038985    6014 scope.go:122] "RemoveContainer" containerID="5ecd9f25b19dc016cb76d7f36dff3d8a6314842c1461bf7bd5443714674746fa"
Oct 30 19:19:43.201947 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:43.201915    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"]
Oct 30 19:19:43.207271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:43.207244    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-732a3f-7f95ccdf85-v4pw4"]
Oct 30 19:19:43.476344 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:43.476313    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9a80a2-6c53-40a3-a5a3-44257e705082" path="/var/lib/kubelet/pods/1b9a80a2-6c53-40a3-a5a3-44257e705082/volumes"
Oct 30 19:19:43.476772 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:43.476751    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1d3787-e640-4027-b267-88a3c8a19873" path="/var/lib/kubelet/pods/7c1d3787-e640-4027-b267-88a3c8a19873/volumes"
Oct 30 19:19:50.822853 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:19:50.822817    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:19:53.259781 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:19:53.259745    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:20:05.514623 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:05.514545    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"]
Oct 30 19:20:12.226365 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:12.226328    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"]
Oct 30 19:20:16.212849 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:16.212809    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:20:17.226948 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:17.226893    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps" podUID="33c08345-1df9-4df2-883d-8715226336b3" containerName="payload-processing" containerID="cri-o://02f6a1d4ed248c951e99d4a69e4a903f5a8caf786ee0f8ed0c3a8041a47f2e3d" gracePeriod=25
Oct 30 19:20:20.536205 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.536123    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" containerID="cri-o://662222cc5bec55f92cfae55c3cc02324f61296564a817d3e106817b1eb7e34dd" gracePeriod=45
Oct 30 19:20:20.617366 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.617320    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 19:20:20.811923 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.811896    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:20:20.940837 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.940800    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-home\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-home\") pod \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") "
Oct 30 19:20:20.941023 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.940881    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tmp-dir\") pod \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") "
Oct 30 19:20:20.941085 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.941069    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tls-certs\") pod \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") "
Oct 30 19:20:20.941133 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.941088    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-home" pod "9c84976c-cb92-4436-99a4-d93ef5cd07d4" (UID: "9c84976c-cb92-4436-99a4-d93ef5cd07d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:20.941133 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.941121    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c84976c-cb92-4436-99a4-d93ef5cd07d4-kube-api-access-6xtk7\" (UniqueName: \"kubernetes.io/projected/9c84976c-cb92-4436-99a4-d93ef5cd07d4-kube-api-access-6xtk7\") pod \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") "
Oct 30 19:20:20.941133 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.941153    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tmp-dir" pod "9c84976c-cb92-4436-99a4-d93ef5cd07d4" (UID: "9c84976c-cb92-4436-99a4-d93ef5cd07d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:20.941276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.941205    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-dshm\") pod \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") "
Oct 30 19:20:20.941315 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.941291    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-model-cache\") pod \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\" (UID: \"9c84976c-cb92-4436-99a4-d93ef5cd07d4\") "
Oct 30 19:20:20.941612 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.941576    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-model-cache" pod "9c84976c-cb92-4436-99a4-d93ef5cd07d4" (UID: "9c84976c-cb92-4436-99a4-d93ef5cd07d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:20.941980 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.941962    6014 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tmp-dir\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:20.942065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.941986    6014 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-model-cache\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:20.942065 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.942003    6014 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-home\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:20.943932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.943905    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c84976c-cb92-4436-99a4-d93ef5cd07d4-kube-api-access-6xtk7" pod "9c84976c-cb92-4436-99a4-d93ef5cd07d4" (UID: "9c84976c-cb92-4436-99a4-d93ef5cd07d4"). InnerVolumeSpecName "kube-api-access-6xtk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:20.944090 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.944068    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-dshm" pod "9c84976c-cb92-4436-99a4-d93ef5cd07d4" (UID: "9c84976c-cb92-4436-99a4-d93ef5cd07d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:20.944090 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:20.944080    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tls-certs" pod "9c84976c-cb92-4436-99a4-d93ef5cd07d4" (UID: "9c84976c-cb92-4436-99a4-d93ef5cd07d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:21.043802 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:21.043703    6014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c84976c-cb92-4436-99a4-d93ef5cd07d4-tls-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:21.043802 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:21.043738    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xtk7\" (UniqueName: \"kubernetes.io/projected/9c84976c-cb92-4436-99a4-d93ef5cd07d4-kube-api-access-6xtk7\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:21.043802 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:21.043748    6014 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c84976c-cb92-4436-99a4-d93ef5cd07d4-dshm\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:21.266898 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:21.266867    6014 generic.go:356] "Generic (PLEG): container finished" podID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerID="662222cc5bec55f92cfae55c3cc02324f61296564a817d3e106817b1eb7e34dd" exitCode=0
Oct 30 19:20:21.267092 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:21.266906    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" event={"ID":"9c84976c-cb92-4436-99a4-d93ef5cd07d4","Type":"ContainerDied","Data":"662222cc5bec55f92cfae55c3cc02324f61296564a817d3e106817b1eb7e34dd"}
Oct 30 19:20:21.267092 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:21.266911    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"
Oct 30 19:20:21.267092 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:21.266939    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z" event={"ID":"9c84976c-cb92-4436-99a4-d93ef5cd07d4","Type":"ContainerDied","Data":"caa1e8037569e5211a3b4f3e74c63434c5331b8dfd921fc6f601097df7cfa04a"}
Oct 30 19:20:21.267092 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:21.266956    6014 scope.go:122] "RemoveContainer" containerID="662222cc5bec55f92cfae55c3cc02324f61296564a817d3e106817b1eb7e34dd"
Oct 30 19:20:21.277660 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:21.277638    6014 scope.go:122] "RemoveContainer" containerID="662222cc5bec55f92cfae55c3cc02324f61296564a817d3e106817b1eb7e34dd"
Oct 30 19:20:21.277950 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:20:21.277929    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662222cc5bec55f92cfae55c3cc02324f61296564a817d3e106817b1eb7e34dd\": container with ID starting with 662222cc5bec55f92cfae55c3cc02324f61296564a817d3e106817b1eb7e34dd not found: ID does not exist" containerID="662222cc5bec55f92cfae55c3cc02324f61296564a817d3e106817b1eb7e34dd"
Oct 30 19:20:21.278019 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:21.277963    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662222cc5bec55f92cfae55c3cc02324f61296564a817d3e106817b1eb7e34dd"} err="failed to get container status \"662222cc5bec55f92cfae55c3cc02324f61296564a817d3e106817b1eb7e34dd\": rpc error: code = NotFound desc = could not find container \"662222cc5bec55f92cfae55c3cc02324f61296564a817d3e106817b1eb7e34dd\": container with ID starting with 662222cc5bec55f92cfae55c3cc02324f61296564a817d3e106817b1eb7e34dd not found: ID does not exist"
Oct 30 19:20:21.431875 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:21.431841    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"]
Oct 30 19:20:21.480319 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:21.480279    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bc98e1/simulator-w3-bc98e1-kserve-76bc58878-bz45z"]
Oct 30 19:20:23.258639 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:20:23.258600    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:20:23.342261 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:23.342223    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps" podUID="33c08345-1df9-4df2-883d-8715226336b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:23.475941 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:23.475907    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" path="/var/lib/kubelet/pods/9c84976c-cb92-4436-99a4-d93ef5cd07d4/volumes"
Oct 30 19:20:25.714765 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:25.714729    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"]
Oct 30 19:20:25.715216 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:25.714940    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" podUID="945375bc-250a-4b52-8cfc-e405096baf26" containerName="istio-proxy" containerID="cri-o://139b673c801b033b8b72f17bbfa823fcd6b1c91daf108eacadc638bc119990c4" gracePeriod=30
Oct 30 19:20:30.983495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:30.983435    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 19:20:31.064030 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.063995    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-kube-api-access-vwp5v\" (UniqueName: \"kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-kube-api-access-vwp5v\") pod \"945375bc-250a-4b52-8cfc-e405096baf26\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") "
Oct 30 19:20:31.064255 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.064098    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/945375bc-250a-4b52-8cfc-e405096baf26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945375bc-250a-4b52-8cfc-e405096baf26-istio-podinfo\") pod \"945375bc-250a-4b52-8cfc-e405096baf26\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") "
Oct 30 19:20:31.064255 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.064195    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-data\") pod \"945375bc-250a-4b52-8cfc-e405096baf26\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") "
Oct 30 19:20:31.064255 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.064230    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-certs\") pod \"945375bc-250a-4b52-8cfc-e405096baf26\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") "
Oct 30 19:20:31.064387 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.064287    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-envoy\") pod \"945375bc-250a-4b52-8cfc-e405096baf26\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") "
Oct 30 19:20:31.064387 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.064319    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945375bc-250a-4b52-8cfc-e405096baf26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945375bc-250a-4b52-8cfc-e405096baf26-istiod-ca-cert\") pod \"945375bc-250a-4b52-8cfc-e405096baf26\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") "
Oct 30 19:20:31.064387 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.064369    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-credential-socket\") pod \"945375bc-250a-4b52-8cfc-e405096baf26\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") "
Oct 30 19:20:31.064544 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.064423    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-istio-token\" (UniqueName: \"kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-istio-token\") pod \"945375bc-250a-4b52-8cfc-e405096baf26\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") "
Oct 30 19:20:31.064544 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.064461    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-socket\") pod \"945375bc-250a-4b52-8cfc-e405096baf26\" (UID: \"945375bc-250a-4b52-8cfc-e405096baf26\") "
Oct 30 19:20:31.064544 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.064526    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-data" pod "945375bc-250a-4b52-8cfc-e405096baf26" (UID: "945375bc-250a-4b52-8cfc-e405096baf26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.064715 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.064621    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-certs" pod "945375bc-250a-4b52-8cfc-e405096baf26" (UID: "945375bc-250a-4b52-8cfc-e405096baf26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.064779 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.064736    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-credential-socket" pod "945375bc-250a-4b52-8cfc-e405096baf26" (UID: "945375bc-250a-4b52-8cfc-e405096baf26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.064779 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.064767    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945375bc-250a-4b52-8cfc-e405096baf26-istiod-ca-cert" pod "945375bc-250a-4b52-8cfc-e405096baf26" (UID: "945375bc-250a-4b52-8cfc-e405096baf26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:31.064906 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.064887    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-socket" pod "945375bc-250a-4b52-8cfc-e405096baf26" (UID: "945375bc-250a-4b52-8cfc-e405096baf26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.065054 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.065033    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.065054 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.065056    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.065282 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.065069    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945375bc-250a-4b52-8cfc-e405096baf26-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.065282 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.065081    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.065282 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.065096    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.066869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.066848    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-envoy" pod "945375bc-250a-4b52-8cfc-e405096baf26" (UID: "945375bc-250a-4b52-8cfc-e405096baf26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.066869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.066854    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-istio-token" pod "945375bc-250a-4b52-8cfc-e405096baf26" (UID: "945375bc-250a-4b52-8cfc-e405096baf26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:31.067108 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.067091    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/945375bc-250a-4b52-8cfc-e405096baf26-istio-podinfo" pod "945375bc-250a-4b52-8cfc-e405096baf26" (UID: "945375bc-250a-4b52-8cfc-e405096baf26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:31.067189 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.067120    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-kube-api-access-vwp5v" pod "945375bc-250a-4b52-8cfc-e405096baf26" (UID: "945375bc-250a-4b52-8cfc-e405096baf26"). InnerVolumeSpecName "kube-api-access-vwp5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:31.165977 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.165935    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwp5v\" (UniqueName: \"kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-kube-api-access-vwp5v\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.165977 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.165969    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945375bc-250a-4b52-8cfc-e405096baf26-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.165977 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.165983    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945375bc-250a-4b52-8cfc-e405096baf26-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.166262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.165995    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945375bc-250a-4b52-8cfc-e405096baf26-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.327043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.327007    6014 generic.go:356] "Generic (PLEG): container finished" podID="945375bc-250a-4b52-8cfc-e405096baf26" containerID="139b673c801b033b8b72f17bbfa823fcd6b1c91daf108eacadc638bc119990c4" exitCode=0
Oct 30 19:20:31.327246 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.327055    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"
Oct 30 19:20:31.327246 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.327074    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" event={"ID":"945375bc-250a-4b52-8cfc-e405096baf26","Type":"ContainerDied","Data":"139b673c801b033b8b72f17bbfa823fcd6b1c91daf108eacadc638bc119990c4"}
Oct 30 19:20:31.327246 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.327119    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t" event={"ID":"945375bc-250a-4b52-8cfc-e405096baf26","Type":"ContainerDied","Data":"aab3ffbbb99437e8d53ed1c15be1f645d7a6fd3c5f492da531978b4d378079c8"}
Oct 30 19:20:31.327246 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.327171    6014 scope.go:122] "RemoveContainer" containerID="139b673c801b033b8b72f17bbfa823fcd6b1c91daf108eacadc638bc119990c4"
Oct 30 19:20:31.337411 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.337392    6014 scope.go:122] "RemoveContainer" containerID="a42f5e791085fd50c3401e21930ab7422508d5b44609144db981f97a1363bd7e"
Oct 30 19:20:31.346357 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.346336    6014 scope.go:122] "RemoveContainer" containerID="139b673c801b033b8b72f17bbfa823fcd6b1c91daf108eacadc638bc119990c4"
Oct 30 19:20:31.346626 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:20:31.346604    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139b673c801b033b8b72f17bbfa823fcd6b1c91daf108eacadc638bc119990c4\": container with ID starting with 139b673c801b033b8b72f17bbfa823fcd6b1c91daf108eacadc638bc119990c4 not found: ID does not exist" containerID="139b673c801b033b8b72f17bbfa823fcd6b1c91daf108eacadc638bc119990c4"
Oct 30 19:20:31.346676 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.346636    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139b673c801b033b8b72f17bbfa823fcd6b1c91daf108eacadc638bc119990c4"} err="failed to get container status \"139b673c801b033b8b72f17bbfa823fcd6b1c91daf108eacadc638bc119990c4\": rpc error: code = NotFound desc = could not find container \"139b673c801b033b8b72f17bbfa823fcd6b1c91daf108eacadc638bc119990c4\": container with ID starting with 139b673c801b033b8b72f17bbfa823fcd6b1c91daf108eacadc638bc119990c4 not found: ID does not exist"
Oct 30 19:20:31.346676 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.346655    6014 scope.go:122] "RemoveContainer" containerID="a42f5e791085fd50c3401e21930ab7422508d5b44609144db981f97a1363bd7e"
Oct 30 19:20:31.346928 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:20:31.346914    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42f5e791085fd50c3401e21930ab7422508d5b44609144db981f97a1363bd7e\": container with ID starting with a42f5e791085fd50c3401e21930ab7422508d5b44609144db981f97a1363bd7e not found: ID does not exist" containerID="a42f5e791085fd50c3401e21930ab7422508d5b44609144db981f97a1363bd7e"
Oct 30 19:20:31.346975 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.346934    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42f5e791085fd50c3401e21930ab7422508d5b44609144db981f97a1363bd7e"} err="failed to get container status \"a42f5e791085fd50c3401e21930ab7422508d5b44609144db981f97a1363bd7e\": rpc error: code = NotFound desc = could not find container \"a42f5e791085fd50c3401e21930ab7422508d5b44609144db981f97a1363bd7e\": container with ID starting with a42f5e791085fd50c3401e21930ab7422508d5b44609144db981f97a1363bd7e not found: ID does not exist"
Oct 30 19:20:31.493288 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.493251    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"]
Oct 30 19:20:31.495131 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:31.495098    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-hr88t"]
Oct 30 19:20:33.342389 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:33.342348    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps" podUID="33c08345-1df9-4df2-883d-8715226336b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:33.475901 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:33.475863    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945375bc-250a-4b52-8cfc-e405096baf26" path="/var/lib/kubelet/pods/945375bc-250a-4b52-8cfc-e405096baf26/volumes"
Oct 30 19:20:33.524705 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:33.524661    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:20:37.253995 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.253962    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"]
Oct 30 19:20:37.254394 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.254209    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv" podUID="1456cdbd-5d69-4731-9da9-6b2e803be1f9" containerName="maas-api" containerID="cri-o://66edfd02157e2136f15f3bb7ecb6d8176e86c77cb421afc5fa6ed1df37adcf3e" gracePeriod=30
Oct 30 19:20:37.364623 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.364591    6014 generic.go:356] "Generic (PLEG): container finished" podID="33c08345-1df9-4df2-883d-8715226336b3" containerID="02f6a1d4ed248c951e99d4a69e4a903f5a8caf786ee0f8ed0c3a8041a47f2e3d" exitCode=0
Oct 30 19:20:37.364732 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.364658    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps" event={"ID":"33c08345-1df9-4df2-883d-8715226336b3","Type":"ContainerDied","Data":"02f6a1d4ed248c951e99d4a69e4a903f5a8caf786ee0f8ed0c3a8041a47f2e3d"}
Oct 30 19:20:37.541220 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.541188    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"
Oct 30 19:20:37.564297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.564274    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 19:20:37.643048 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.643015    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c08345-1df9-4df2-883d-8715226336b3-kube-api-access-j7b5s\" (UniqueName: \"kubernetes.io/projected/33c08345-1df9-4df2-883d-8715226336b3-kube-api-access-j7b5s\") pod \"33c08345-1df9-4df2-883d-8715226336b3\" (UID: \"33c08345-1df9-4df2-883d-8715226336b3\") "
Oct 30 19:20:37.643269 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.643155    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c08345-1df9-4df2-883d-8715226336b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c08345-1df9-4df2-883d-8715226336b3-plugins-config-volume\") pod \"33c08345-1df9-4df2-883d-8715226336b3\" (UID: \"33c08345-1df9-4df2-883d-8715226336b3\") "
Oct 30 19:20:37.643557 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.643534    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c08345-1df9-4df2-883d-8715226336b3-plugins-config-volume" pod "33c08345-1df9-4df2-883d-8715226336b3" (UID: "33c08345-1df9-4df2-883d-8715226336b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:37.643661 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.643626    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c08345-1df9-4df2-883d-8715226336b3-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:37.646457 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.646357    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c08345-1df9-4df2-883d-8715226336b3-kube-api-access-j7b5s" pod "33c08345-1df9-4df2-883d-8715226336b3" (UID: "33c08345-1df9-4df2-883d-8715226336b3"). InnerVolumeSpecName "kube-api-access-j7b5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:37.744950 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.744855    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-maas-api-tls\") pod \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") "
Oct 30 19:20:37.744950 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.744907    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1456cdbd-5d69-4731-9da9-6b2e803be1f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1456cdbd-5d69-4731-9da9-6b2e803be1f9-openshift-service-ca\") pod \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") "
Oct 30 19:20:37.745245 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.745003    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-metrics-certs\") pod \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") "
Oct 30 19:20:37.745245 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.745179    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1456cdbd-5d69-4731-9da9-6b2e803be1f9-kube-api-access-btdnj\" (UniqueName: \"kubernetes.io/projected/1456cdbd-5d69-4731-9da9-6b2e803be1f9-kube-api-access-btdnj\") pod \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\" (UID: \"1456cdbd-5d69-4731-9da9-6b2e803be1f9\") "
Oct 30 19:20:37.745401 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.745370    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1456cdbd-5d69-4731-9da9-6b2e803be1f9-openshift-service-ca" pod "1456cdbd-5d69-4731-9da9-6b2e803be1f9" (UID: "1456cdbd-5d69-4731-9da9-6b2e803be1f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:37.745685 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.745662    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7b5s\" (UniqueName: \"kubernetes.io/projected/33c08345-1df9-4df2-883d-8715226336b3-kube-api-access-j7b5s\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:37.745782 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.745689    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1456cdbd-5d69-4731-9da9-6b2e803be1f9-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:37.747336 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.747310    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-metrics-certs" pod "1456cdbd-5d69-4731-9da9-6b2e803be1f9" (UID: "1456cdbd-5d69-4731-9da9-6b2e803be1f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:37.747443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.747333    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-maas-api-tls" pod "1456cdbd-5d69-4731-9da9-6b2e803be1f9" (UID: "1456cdbd-5d69-4731-9da9-6b2e803be1f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:37.747443 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.747339    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1456cdbd-5d69-4731-9da9-6b2e803be1f9-kube-api-access-btdnj" pod "1456cdbd-5d69-4731-9da9-6b2e803be1f9" (UID: "1456cdbd-5d69-4731-9da9-6b2e803be1f9"). InnerVolumeSpecName "kube-api-access-btdnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:37.846658 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.846626    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btdnj\" (UniqueName: \"kubernetes.io/projected/1456cdbd-5d69-4731-9da9-6b2e803be1f9-kube-api-access-btdnj\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:37.846658 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.846655    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:37.846658 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:37.846667    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1456cdbd-5d69-4731-9da9-6b2e803be1f9-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:38.371496 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.371459    6014 generic.go:356] "Generic (PLEG): container finished" podID="1456cdbd-5d69-4731-9da9-6b2e803be1f9" containerID="66edfd02157e2136f15f3bb7ecb6d8176e86c77cb421afc5fa6ed1df37adcf3e" exitCode=0
Oct 30 19:20:38.371984 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.371506    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"
Oct 30 19:20:38.371984 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.371544    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv" event={"ID":"1456cdbd-5d69-4731-9da9-6b2e803be1f9","Type":"ContainerDied","Data":"66edfd02157e2136f15f3bb7ecb6d8176e86c77cb421afc5fa6ed1df37adcf3e"}
Oct 30 19:20:38.371984 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.371597    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv" event={"ID":"1456cdbd-5d69-4731-9da9-6b2e803be1f9","Type":"ContainerDied","Data":"ec08c99c93f4802bf701d850d61353a6496c2799ae19fa7616888ec59138caac"}
Oct 30 19:20:38.371984 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.371623    6014 scope.go:122] "RemoveContainer" containerID="66edfd02157e2136f15f3bb7ecb6d8176e86c77cb421afc5fa6ed1df37adcf3e"
Oct 30 19:20:38.373285 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.373265    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps" event={"ID":"33c08345-1df9-4df2-883d-8715226336b3","Type":"ContainerDied","Data":"7c9132041741b9daa540c6207450c5cdf0922a3c5bbaf235e83845bb226f24e6"}
Oct 30 19:20:38.373379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.373294    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"
Oct 30 19:20:38.384283 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.384265    6014 scope.go:122] "RemoveContainer" containerID="66edfd02157e2136f15f3bb7ecb6d8176e86c77cb421afc5fa6ed1df37adcf3e"
Oct 30 19:20:38.384582 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:20:38.384565    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66edfd02157e2136f15f3bb7ecb6d8176e86c77cb421afc5fa6ed1df37adcf3e\": container with ID starting with 66edfd02157e2136f15f3bb7ecb6d8176e86c77cb421afc5fa6ed1df37adcf3e not found: ID does not exist" containerID="66edfd02157e2136f15f3bb7ecb6d8176e86c77cb421afc5fa6ed1df37adcf3e"
Oct 30 19:20:38.384640 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.384588    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66edfd02157e2136f15f3bb7ecb6d8176e86c77cb421afc5fa6ed1df37adcf3e"} err="failed to get container status \"66edfd02157e2136f15f3bb7ecb6d8176e86c77cb421afc5fa6ed1df37adcf3e\": rpc error: code = NotFound desc = could not find container \"66edfd02157e2136f15f3bb7ecb6d8176e86c77cb421afc5fa6ed1df37adcf3e\": container with ID starting with 66edfd02157e2136f15f3bb7ecb6d8176e86c77cb421afc5fa6ed1df37adcf3e not found: ID does not exist"
Oct 30 19:20:38.384640 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.384605    6014 scope.go:122] "RemoveContainer" containerID="02f6a1d4ed248c951e99d4a69e4a903f5a8caf786ee0f8ed0c3a8041a47f2e3d"
Oct 30 19:20:38.538002 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.537912    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"]
Oct 30 19:20:38.590129 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.590082    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bc98e1-5467bb875c-wcvps"]
Oct 30 19:20:38.650480 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.650442    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"]
Oct 30 19:20:38.653817 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:38.653792    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bc98e1-d75bfbf4c-8s4zv"]
Oct 30 19:20:39.475732 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:39.475699    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1456cdbd-5d69-4731-9da9-6b2e803be1f9" path="/var/lib/kubelet/pods/1456cdbd-5d69-4731-9da9-6b2e803be1f9/volumes"
Oct 30 19:20:39.476155 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:39.476126    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c08345-1df9-4df2-883d-8715226336b3" path="/var/lib/kubelet/pods/33c08345-1df9-4df2-883d-8715226336b3/volumes"
Oct 30 19:20:53.258124 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:20:53.258083    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:20:59.425419 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:20:59.425383    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:21:14.802013 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:14.801976    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"]
Oct 30 19:21:20.343709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:20.343675    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:21:23.258298 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:21:23.258260    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:21:29.822854 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:29.822796    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll" podUID="da64e353-542e-4e5a-8a31-ef6c1891e5f8" containerName="main" containerID="cri-o://0bb09bd015ced379c4ca439b9e3cd0cc645df647da45043b41bcabf5cde2be1b" gracePeriod=45
Oct 30 19:21:30.096822 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.096795    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:21:30.245717 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.245682    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tmp-dir\") pod \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") "
Oct 30 19:21:30.245717 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.245725    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tls-certs\") pod \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") "
Oct 30 19:21:30.245990 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.245815    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-dshm\") pod \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") "
Oct 30 19:21:30.245990 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.245887    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-model-cache\") pod \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") "
Oct 30 19:21:30.245990 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.245985    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-home\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-home\") pod \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") "
Oct 30 19:21:30.246183 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.245990    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tmp-dir" pod "da64e353-542e-4e5a-8a31-ef6c1891e5f8" (UID: "da64e353-542e-4e5a-8a31-ef6c1891e5f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:30.246183 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.246065    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da64e353-542e-4e5a-8a31-ef6c1891e5f8-kube-api-access-b8rq4\" (UniqueName: \"kubernetes.io/projected/da64e353-542e-4e5a-8a31-ef6c1891e5f8-kube-api-access-b8rq4\") pod \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\" (UID: \"da64e353-542e-4e5a-8a31-ef6c1891e5f8\") "
Oct 30 19:21:30.246303 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.246211    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-model-cache" pod "da64e353-542e-4e5a-8a31-ef6c1891e5f8" (UID: "da64e353-542e-4e5a-8a31-ef6c1891e5f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:30.246382 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.246362    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-home" pod "da64e353-542e-4e5a-8a31-ef6c1891e5f8" (UID: "da64e353-542e-4e5a-8a31-ef6c1891e5f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:30.246553 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.246530    6014 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-home\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:30.246720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.246556    6014 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tmp-dir\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:30.246720 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.246572    6014 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-model-cache\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:30.249225 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.249166    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tls-certs" pod "da64e353-542e-4e5a-8a31-ef6c1891e5f8" (UID: "da64e353-542e-4e5a-8a31-ef6c1891e5f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:30.249225 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.249176    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-dshm" pod "da64e353-542e-4e5a-8a31-ef6c1891e5f8" (UID: "da64e353-542e-4e5a-8a31-ef6c1891e5f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:30.249398 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.249281    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da64e353-542e-4e5a-8a31-ef6c1891e5f8-kube-api-access-b8rq4" pod "da64e353-542e-4e5a-8a31-ef6c1891e5f8" (UID: "da64e353-542e-4e5a-8a31-ef6c1891e5f8"). InnerVolumeSpecName "kube-api-access-b8rq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:30.347838 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.347739    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8rq4\" (UniqueName: \"kubernetes.io/projected/da64e353-542e-4e5a-8a31-ef6c1891e5f8-kube-api-access-b8rq4\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:30.347838 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.347780    6014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da64e353-542e-4e5a-8a31-ef6c1891e5f8-tls-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:30.347838 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.347794    6014 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da64e353-542e-4e5a-8a31-ef6c1891e5f8-dshm\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:30.677312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.677181    6014 generic.go:356] "Generic (PLEG): container finished" podID="da64e353-542e-4e5a-8a31-ef6c1891e5f8" containerID="0bb09bd015ced379c4ca439b9e3cd0cc645df647da45043b41bcabf5cde2be1b" exitCode=0
Oct 30 19:21:30.677312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.677235    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"
Oct 30 19:21:30.677312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.677275    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll" event={"ID":"da64e353-542e-4e5a-8a31-ef6c1891e5f8","Type":"ContainerDied","Data":"0bb09bd015ced379c4ca439b9e3cd0cc645df647da45043b41bcabf5cde2be1b"}
Oct 30 19:21:30.677683 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.677330    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll" event={"ID":"da64e353-542e-4e5a-8a31-ef6c1891e5f8","Type":"ContainerDied","Data":"135650686ee46c9e99a237c08386e4cf71d8941ae49c6a95a5dc7902fdddf23d"}
Oct 30 19:21:30.677683 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.677353    6014 scope.go:122] "RemoveContainer" containerID="0bb09bd015ced379c4ca439b9e3cd0cc645df647da45043b41bcabf5cde2be1b"
Oct 30 19:21:30.692211 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.692190    6014 scope.go:122] "RemoveContainer" containerID="0bb09bd015ced379c4ca439b9e3cd0cc645df647da45043b41bcabf5cde2be1b"
Oct 30 19:21:30.692519 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:21:30.692498    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb09bd015ced379c4ca439b9e3cd0cc645df647da45043b41bcabf5cde2be1b\": container with ID starting with 0bb09bd015ced379c4ca439b9e3cd0cc645df647da45043b41bcabf5cde2be1b not found: ID does not exist" containerID="0bb09bd015ced379c4ca439b9e3cd0cc645df647da45043b41bcabf5cde2be1b"
Oct 30 19:21:30.692584 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.692526    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb09bd015ced379c4ca439b9e3cd0cc645df647da45043b41bcabf5cde2be1b"} err="failed to get container status \"0bb09bd015ced379c4ca439b9e3cd0cc645df647da45043b41bcabf5cde2be1b\": rpc error: code = NotFound desc = could not find container \"0bb09bd015ced379c4ca439b9e3cd0cc645df647da45043b41bcabf5cde2be1b\": container with ID starting with 0bb09bd015ced379c4ca439b9e3cd0cc645df647da45043b41bcabf5cde2be1b not found: ID does not exist"
Oct 30 19:21:30.844785 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.844751    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"]
Oct 30 19:21:30.895515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:30.895474    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e0fc1f/test-model-e0fc1f-kserve-b797b84f-xxbll"]
Oct 30 19:21:31.476354 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:31.476319    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da64e353-542e-4e5a-8a31-ef6c1891e5f8" path="/var/lib/kubelet/pods/da64e353-542e-4e5a-8a31-ef6c1891e5f8/volumes"
Oct 30 19:21:33.134390 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:33.134355    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:21:43.290793 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:43.290758    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"]
Oct 30 19:21:49.810385 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:49.810350    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:21:53.260101 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:21:53.260061    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:21:54.893686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:54.893652    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"]
Oct 30 19:21:54.894118 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:54.893871    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk" podUID="a99523a9-e39a-4c4d-b06d-9e27c847ffd5" containerName="maas-api" containerID="cri-o://d6347e4aec71cf935a5506c0dbc0bd5518f37de8c7e8e0084f7f63fa3feb281e" gracePeriod=30
Oct 30 19:21:55.152253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.152183    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:21:55.224298 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.224262    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-maas-api-tls\") pod \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") "
Oct 30 19:21:55.224475 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.224324    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-metrics-certs\") pod \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") "
Oct 30 19:21:55.224475 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.224363    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-openshift-service-ca\") pod \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") "
Oct 30 19:21:55.224475 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.224469    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-kube-api-access-tr5b8\" (UniqueName: \"kubernetes.io/projected/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-kube-api-access-tr5b8\") pod \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\" (UID: \"a99523a9-e39a-4c4d-b06d-9e27c847ffd5\") "
Oct 30 19:21:55.224806 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.224774    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-openshift-service-ca" pod "a99523a9-e39a-4c4d-b06d-9e27c847ffd5" (UID: "a99523a9-e39a-4c4d-b06d-9e27c847ffd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:55.226748 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.226722    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-metrics-certs" pod "a99523a9-e39a-4c4d-b06d-9e27c847ffd5" (UID: "a99523a9-e39a-4c4d-b06d-9e27c847ffd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:55.226860 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.226763    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-maas-api-tls" pod "a99523a9-e39a-4c4d-b06d-9e27c847ffd5" (UID: "a99523a9-e39a-4c4d-b06d-9e27c847ffd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:55.226860 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.226819    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-kube-api-access-tr5b8" pod "a99523a9-e39a-4c4d-b06d-9e27c847ffd5" (UID: "a99523a9-e39a-4c4d-b06d-9e27c847ffd5"). InnerVolumeSpecName "kube-api-access-tr5b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:55.325811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.325777    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:55.325811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.325803    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:55.325811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.325813    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:55.326063 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.325822    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr5b8\" (UniqueName: \"kubernetes.io/projected/a99523a9-e39a-4c4d-b06d-9e27c847ffd5-kube-api-access-tr5b8\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:55.845874 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.845831    6014 generic.go:356] "Generic (PLEG): container finished" podID="a99523a9-e39a-4c4d-b06d-9e27c847ffd5" containerID="d6347e4aec71cf935a5506c0dbc0bd5518f37de8c7e8e0084f7f63fa3feb281e" exitCode=0
Oct 30 19:21:55.846103 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.845944    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"
Oct 30 19:21:55.846264 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.845948    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk" event={"ID":"a99523a9-e39a-4c4d-b06d-9e27c847ffd5","Type":"ContainerDied","Data":"d6347e4aec71cf935a5506c0dbc0bd5518f37de8c7e8e0084f7f63fa3feb281e"}
Oct 30 19:21:55.846360 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.846332    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk" event={"ID":"a99523a9-e39a-4c4d-b06d-9e27c847ffd5","Type":"ContainerDied","Data":"470842a9bc9ef07b509581a2ee70bb6ad602e8f8ff5533bfb231ee3776fb50b2"}
Oct 30 19:21:55.846427 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.846365    6014 scope.go:122] "RemoveContainer" containerID="d6347e4aec71cf935a5506c0dbc0bd5518f37de8c7e8e0084f7f63fa3feb281e"
Oct 30 19:21:55.857545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.857519    6014 scope.go:122] "RemoveContainer" containerID="d6347e4aec71cf935a5506c0dbc0bd5518f37de8c7e8e0084f7f63fa3feb281e"
Oct 30 19:21:55.857901 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:21:55.857874    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6347e4aec71cf935a5506c0dbc0bd5518f37de8c7e8e0084f7f63fa3feb281e\": container with ID starting with d6347e4aec71cf935a5506c0dbc0bd5518f37de8c7e8e0084f7f63fa3feb281e not found: ID does not exist" containerID="d6347e4aec71cf935a5506c0dbc0bd5518f37de8c7e8e0084f7f63fa3feb281e"
Oct 30 19:21:55.858036 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:55.857912    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6347e4aec71cf935a5506c0dbc0bd5518f37de8c7e8e0084f7f63fa3feb281e"} err="failed to get container status \"d6347e4aec71cf935a5506c0dbc0bd5518f37de8c7e8e0084f7f63fa3feb281e\": rpc error: code = NotFound desc = could not find container \"d6347e4aec71cf935a5506c0dbc0bd5518f37de8c7e8e0084f7f63fa3feb281e\": container with ID starting with d6347e4aec71cf935a5506c0dbc0bd5518f37de8c7e8e0084f7f63fa3feb281e not found: ID does not exist"
Oct 30 19:21:56.007948 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:56.007907    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"]
Oct 30 19:21:56.013593 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:56.013557    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e0fc1f-7696bf874d-pjdmk"]
Oct 30 19:21:57.475366 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:57.475333    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99523a9-e39a-4c4d-b06d-9e27c847ffd5" path="/var/lib/kubelet/pods/a99523a9-e39a-4c4d-b06d-9e27c847ffd5/volumes"
Oct 30 19:21:58.311595 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.311551    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q" podUID="1752af1b-aa7b-4c37-8e5d-0ebc54622df0" containerName="main" containerID="cri-o://cfe05ceb1996c09d8355166eeb83854932b89beb08c4ddad438194fbf6fc6d6c" gracePeriod=45
Oct 30 19:21:58.578416 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.578391    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:21:58.671987 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.671945    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tmp-dir\") pod \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") "
Oct 30 19:21:58.672195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.672018    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-model-cache\") pod \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") "
Oct 30 19:21:58.672195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.672072    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-kube-api-access-mt94t\" (UniqueName: \"kubernetes.io/projected/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-kube-api-access-mt94t\") pod \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") "
Oct 30 19:21:58.672195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.672103    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-home\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-home\") pod \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") "
Oct 30 19:21:58.672195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.672185    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tls-certs\" (UniqueName: \"kubernetes.io/secret/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tls-certs\") pod \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") "
Oct 30 19:21:58.672453 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.672356    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-dshm\") pod \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\" (UID: \"1752af1b-aa7b-4c37-8e5d-0ebc54622df0\") "
Oct 30 19:21:58.672453 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.672332    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tmp-dir" pod "1752af1b-aa7b-4c37-8e5d-0ebc54622df0" (UID: "1752af1b-aa7b-4c37-8e5d-0ebc54622df0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:58.672453 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.672371    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-model-cache" pod "1752af1b-aa7b-4c37-8e5d-0ebc54622df0" (UID: "1752af1b-aa7b-4c37-8e5d-0ebc54622df0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:58.672453 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.672433    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-home" pod "1752af1b-aa7b-4c37-8e5d-0ebc54622df0" (UID: "1752af1b-aa7b-4c37-8e5d-0ebc54622df0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:58.672735 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.672712    6014 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tmp-dir\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:58.673032 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.672736    6014 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-model-cache\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:58.673032 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.672752    6014 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-home\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:58.674633 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.674609    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-kube-api-access-mt94t" pod "1752af1b-aa7b-4c37-8e5d-0ebc54622df0" (UID: "1752af1b-aa7b-4c37-8e5d-0ebc54622df0"). InnerVolumeSpecName "kube-api-access-mt94t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:58.674732 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.674648    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-dshm" pod "1752af1b-aa7b-4c37-8e5d-0ebc54622df0" (UID: "1752af1b-aa7b-4c37-8e5d-0ebc54622df0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:58.674732 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.674648    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tls-certs" pod "1752af1b-aa7b-4c37-8e5d-0ebc54622df0" (UID: "1752af1b-aa7b-4c37-8e5d-0ebc54622df0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:58.773664 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.773626    6014 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-dshm\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:58.773664 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.773659    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt94t\" (UniqueName: \"kubernetes.io/projected/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-kube-api-access-mt94t\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:58.773664 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.773671    6014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1752af1b-aa7b-4c37-8e5d-0ebc54622df0-tls-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:58.866309 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.866221    6014 generic.go:356] "Generic (PLEG): container finished" podID="1752af1b-aa7b-4c37-8e5d-0ebc54622df0" containerID="cfe05ceb1996c09d8355166eeb83854932b89beb08c4ddad438194fbf6fc6d6c" exitCode=0
Oct 30 19:21:58.866309 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.866270    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"
Oct 30 19:21:58.866522 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.866305    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q" event={"ID":"1752af1b-aa7b-4c37-8e5d-0ebc54622df0","Type":"ContainerDied","Data":"cfe05ceb1996c09d8355166eeb83854932b89beb08c4ddad438194fbf6fc6d6c"}
Oct 30 19:21:58.866522 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.866351    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q" event={"ID":"1752af1b-aa7b-4c37-8e5d-0ebc54622df0","Type":"ContainerDied","Data":"000f5575ececab8367d0831f8925088463649e3e01e2eb61cf9862b020d8476d"}
Oct 30 19:21:58.866522 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.866368    6014 scope.go:122] "RemoveContainer" containerID="cfe05ceb1996c09d8355166eeb83854932b89beb08c4ddad438194fbf6fc6d6c"
Oct 30 19:21:58.878489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.877289    6014 scope.go:122] "RemoveContainer" containerID="cfe05ceb1996c09d8355166eeb83854932b89beb08c4ddad438194fbf6fc6d6c"
Oct 30 19:21:58.878489 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:21:58.877599    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe05ceb1996c09d8355166eeb83854932b89beb08c4ddad438194fbf6fc6d6c\": container with ID starting with cfe05ceb1996c09d8355166eeb83854932b89beb08c4ddad438194fbf6fc6d6c not found: ID does not exist" containerID="cfe05ceb1996c09d8355166eeb83854932b89beb08c4ddad438194fbf6fc6d6c"
Oct 30 19:21:58.878489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:58.877635    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe05ceb1996c09d8355166eeb83854932b89beb08c4ddad438194fbf6fc6d6c"} err="failed to get container status \"cfe05ceb1996c09d8355166eeb83854932b89beb08c4ddad438194fbf6fc6d6c\": rpc error: code = NotFound desc = could not find container \"cfe05ceb1996c09d8355166eeb83854932b89beb08c4ddad438194fbf6fc6d6c\": container with ID starting with cfe05ceb1996c09d8355166eeb83854932b89beb08c4ddad438194fbf6fc6d6c not found: ID does not exist"
Oct 30 19:21:59.031111 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:59.031070    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"]
Oct 30 19:21:59.034597 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:59.034567    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-714b2c/test-model-714b2c-kserve-7c578f7469-8nk2q"]
Oct 30 19:21:59.476787 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:21:59.476758    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1752af1b-aa7b-4c37-8e5d-0ebc54622df0" path="/var/lib/kubelet/pods/1752af1b-aa7b-4c37-8e5d-0ebc54622df0/volumes"
Oct 30 19:22:04.054916 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:04.054883    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:22:19.550320 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:19.550281    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"]
Oct 30 19:22:19.662243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:19.662198    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"]
Oct 30 19:22:23.258641 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:22:23.258603    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:22:24.550982 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:24.550938    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk" podUID="b420a535-af91-4068-ae30-fef541da631b" containerName="payload-processing" containerID="cri-o://e6f4e5d20b3f25bbb6949e61dac3b388408e4a4aa37031d64e8bc09afd786598" gracePeriod=25
Oct 30 19:22:24.662722 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:24.662654    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm" podUID="cedbbcda-3367-4df0-bbca-e9af8db159cd" containerName="payload-pre-processing" containerID="cri-o://0bc8a1e0da64b982aa04f7457f1ce49cc2b7b44b853e800c15328519c93f01de" gracePeriod=25
Oct 30 19:22:24.812997 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:24.812933    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"
Oct 30 19:22:24.962563 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:24.962529    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b420a535-af91-4068-ae30-fef541da631b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b420a535-af91-4068-ae30-fef541da631b-plugins-config-volume\") pod \"b420a535-af91-4068-ae30-fef541da631b\" (UID: \"b420a535-af91-4068-ae30-fef541da631b\") "
Oct 30 19:22:24.962563 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:24.962569    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b420a535-af91-4068-ae30-fef541da631b-kube-api-access-zlzrm\" (UniqueName: \"kubernetes.io/projected/b420a535-af91-4068-ae30-fef541da631b-kube-api-access-zlzrm\") pod \"b420a535-af91-4068-ae30-fef541da631b\" (UID: \"b420a535-af91-4068-ae30-fef541da631b\") "
Oct 30 19:22:24.963015 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:24.962987    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b420a535-af91-4068-ae30-fef541da631b-plugins-config-volume" pod "b420a535-af91-4068-ae30-fef541da631b" (UID: "b420a535-af91-4068-ae30-fef541da631b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:24.963157 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:24.963120    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b420a535-af91-4068-ae30-fef541da631b-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.965009 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:24.964986    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b420a535-af91-4068-ae30-fef541da631b-kube-api-access-zlzrm" pod "b420a535-af91-4068-ae30-fef541da631b" (UID: "b420a535-af91-4068-ae30-fef541da631b"). InnerVolumeSpecName "kube-api-access-zlzrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:25.012342 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.012305    6014 generic.go:356] "Generic (PLEG): container finished" podID="b420a535-af91-4068-ae30-fef541da631b" containerID="e6f4e5d20b3f25bbb6949e61dac3b388408e4a4aa37031d64e8bc09afd786598" exitCode=0
Oct 30 19:22:25.012536 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.012356    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"
Oct 30 19:22:25.012536 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.012388    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk" event={"ID":"b420a535-af91-4068-ae30-fef541da631b","Type":"ContainerDied","Data":"e6f4e5d20b3f25bbb6949e61dac3b388408e4a4aa37031d64e8bc09afd786598"}
Oct 30 19:22:25.012536 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.012469    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk" event={"ID":"b420a535-af91-4068-ae30-fef541da631b","Type":"ContainerDied","Data":"1d2749f61f17742d908d0733ff3f422a71bc669b1cda4655b369252d40d3c8e9"}
Oct 30 19:22:25.012536 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.012492    6014 scope.go:122] "RemoveContainer" containerID="e6f4e5d20b3f25bbb6949e61dac3b388408e4a4aa37031d64e8bc09afd786598"
Oct 30 19:22:25.025254 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.025233    6014 scope.go:122] "RemoveContainer" containerID="e6f4e5d20b3f25bbb6949e61dac3b388408e4a4aa37031d64e8bc09afd786598"
Oct 30 19:22:25.025604 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:22:25.025585    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f4e5d20b3f25bbb6949e61dac3b388408e4a4aa37031d64e8bc09afd786598\": container with ID starting with e6f4e5d20b3f25bbb6949e61dac3b388408e4a4aa37031d64e8bc09afd786598 not found: ID does not exist" containerID="e6f4e5d20b3f25bbb6949e61dac3b388408e4a4aa37031d64e8bc09afd786598"
Oct 30 19:22:25.025671 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.025616    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f4e5d20b3f25bbb6949e61dac3b388408e4a4aa37031d64e8bc09afd786598"} err="failed to get container status \"e6f4e5d20b3f25bbb6949e61dac3b388408e4a4aa37031d64e8bc09afd786598\": rpc error: code = NotFound desc = could not find container \"e6f4e5d20b3f25bbb6949e61dac3b388408e4a4aa37031d64e8bc09afd786598\": container with ID starting with e6f4e5d20b3f25bbb6949e61dac3b388408e4a4aa37031d64e8bc09afd786598 not found: ID does not exist"
Oct 30 19:22:25.064318 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.064288    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlzrm\" (UniqueName: \"kubernetes.io/projected/b420a535-af91-4068-ae30-fef541da631b-kube-api-access-zlzrm\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:25.167186 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.167130    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"]
Oct 30 19:22:25.167399 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.167357    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx" podUID="6d1c8c76-4e8d-4808-a8f4-09bd5e49f831" containerName="maas-api" containerID="cri-o://8594f100a9197a487e719ee79387a35485c522304fca82f1ab62db05a7f78df8" gracePeriod=30
Oct 30 19:22:25.179297 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.179263    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"]
Oct 30 19:22:25.185933 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.185903    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-714b2c-84d7f4c649-pnhdk"]
Oct 30 19:22:25.422123 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.422098    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:22:25.475366 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.475335    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b420a535-af91-4068-ae30-fef541da631b" path="/var/lib/kubelet/pods/b420a535-af91-4068-ae30-fef541da631b/volumes"
Oct 30 19:22:25.569797 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.569566    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-metrics-certs\") pod \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") "
Oct 30 19:22:25.569797 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.569609    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-maas-api-tls\") pod \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") "
Oct 30 19:22:25.569797 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.569719    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-kube-api-access-b9zlz\" (UniqueName: \"kubernetes.io/projected/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-kube-api-access-b9zlz\") pod \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") "
Oct 30 19:22:25.570446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.569813    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-openshift-service-ca\") pod \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\" (UID: \"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831\") "
Oct 30 19:22:25.570668 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.570545    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-openshift-service-ca" pod "6d1c8c76-4e8d-4808-a8f4-09bd5e49f831" (UID: "6d1c8c76-4e8d-4808-a8f4-09bd5e49f831"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:25.571709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.571682    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:25.572464 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.572434    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-metrics-certs" pod "6d1c8c76-4e8d-4808-a8f4-09bd5e49f831" (UID: "6d1c8c76-4e8d-4808-a8f4-09bd5e49f831"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:25.572555 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.572503    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-maas-api-tls" pod "6d1c8c76-4e8d-4808-a8f4-09bd5e49f831" (UID: "6d1c8c76-4e8d-4808-a8f4-09bd5e49f831"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:25.572700 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.572672    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-kube-api-access-b9zlz" pod "6d1c8c76-4e8d-4808-a8f4-09bd5e49f831" (UID: "6d1c8c76-4e8d-4808-a8f4-09bd5e49f831"). InnerVolumeSpecName "kube-api-access-b9zlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:25.673127 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.673030    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9zlz\" (UniqueName: \"kubernetes.io/projected/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-kube-api-access-b9zlz\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:25.673127 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.673061    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:25.673127 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:25.673072    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:26.019704 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:26.019611    6014 generic.go:356] "Generic (PLEG): container finished" podID="6d1c8c76-4e8d-4808-a8f4-09bd5e49f831" containerID="8594f100a9197a487e719ee79387a35485c522304fca82f1ab62db05a7f78df8" exitCode=0
Oct 30 19:22:26.019704 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:26.019659    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"
Oct 30 19:22:26.019704 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:26.019691    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx" event={"ID":"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831","Type":"ContainerDied","Data":"8594f100a9197a487e719ee79387a35485c522304fca82f1ab62db05a7f78df8"}
Oct 30 19:22:26.019977 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:26.019740    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx" event={"ID":"6d1c8c76-4e8d-4808-a8f4-09bd5e49f831","Type":"ContainerDied","Data":"50fbc46ffdfbffa7b698c411f59e09882e6563d72b98319608a2728889c19565"}
Oct 30 19:22:26.019977 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:26.019761    6014 scope.go:122] "RemoveContainer" containerID="8594f100a9197a487e719ee79387a35485c522304fca82f1ab62db05a7f78df8"
Oct 30 19:22:26.029505 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:26.029486    6014 scope.go:122] "RemoveContainer" containerID="8594f100a9197a487e719ee79387a35485c522304fca82f1ab62db05a7f78df8"
Oct 30 19:22:26.029765 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:22:26.029747    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8594f100a9197a487e719ee79387a35485c522304fca82f1ab62db05a7f78df8\": container with ID starting with 8594f100a9197a487e719ee79387a35485c522304fca82f1ab62db05a7f78df8 not found: ID does not exist" containerID="8594f100a9197a487e719ee79387a35485c522304fca82f1ab62db05a7f78df8"
Oct 30 19:22:26.029833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:26.029778    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8594f100a9197a487e719ee79387a35485c522304fca82f1ab62db05a7f78df8"} err="failed to get container status \"8594f100a9197a487e719ee79387a35485c522304fca82f1ab62db05a7f78df8\": rpc error: code = NotFound desc = could not find container \"8594f100a9197a487e719ee79387a35485c522304fca82f1ab62db05a7f78df8\": container with ID starting with 8594f100a9197a487e719ee79387a35485c522304fca82f1ab62db05a7f78df8 not found: ID does not exist"
Oct 30 19:22:26.186105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:26.186069    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"]
Oct 30 19:22:26.189490 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:26.189462    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-714b2c-7cdd88dfc5-94rwx"]
Oct 30 19:22:27.475589 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:27.475555    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d1c8c76-4e8d-4808-a8f4-09bd5e49f831" path="/var/lib/kubelet/pods/6d1c8c76-4e8d-4808-a8f4-09bd5e49f831/volumes"
Oct 30 19:22:30.146779 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:30.146737    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm" podUID="cedbbcda-3367-4df0-bbca-e9af8db159cd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:30.528774 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:30.528674    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:22:35.003768 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.003725    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"]
Oct 30 19:22:35.004260 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.003972    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9" podUID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerName="istio-proxy" containerID="cri-o://d0e12575132ca27aa7dacde8999847ca451cba4646e5b1e0093b1cec125310f0" gracePeriod=30
Oct 30 19:22:35.775484 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.775449    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"]
Oct 30 19:22:35.775937 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.775925    6014 cpu_manager.go:414] "containerMap: removing container" podUID="945375bc-250a-4b52-8cfc-e405096baf26" containerName="istio-proxy"
Oct 30 19:22:35.775988 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.775938    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945375bc-250a-4b52-8cfc-e405096baf26" containerName="istio-proxy"
Oct 30 19:22:35.775988 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.775952    6014 cpu_manager.go:414] "containerMap: removing container" podUID="6d1c8c76-4e8d-4808-a8f4-09bd5e49f831" containerName="maas-api"
Oct 30 19:22:35.775988 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.775957    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1c8c76-4e8d-4808-a8f4-09bd5e49f831" containerName="maas-api"
Oct 30 19:22:35.775988 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.775972    6014 cpu_manager.go:414] "containerMap: removing container" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main"
Oct 30 19:22:35.775988 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.775977    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.775996    6014 cpu_manager.go:414] "containerMap: removing container" podUID="da64e353-542e-4e5a-8a31-ef6c1891e5f8" containerName="main"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776001    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da64e353-542e-4e5a-8a31-ef6c1891e5f8" containerName="main"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776007    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1b9a80a2-6c53-40a3-a5a3-44257e705082" containerName="payload-processing"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776012    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9a80a2-6c53-40a3-a5a3-44257e705082" containerName="payload-processing"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776019    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1456cdbd-5d69-4731-9da9-6b2e803be1f9" containerName="maas-api"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776024    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1456cdbd-5d69-4731-9da9-6b2e803be1f9" containerName="maas-api"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776034    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1752af1b-aa7b-4c37-8e5d-0ebc54622df0" containerName="main"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776038    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1752af1b-aa7b-4c37-8e5d-0ebc54622df0" containerName="main"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776047    6014 cpu_manager.go:414] "containerMap: removing container" podUID="b420a535-af91-4068-ae30-fef541da631b" containerName="payload-processing"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776052    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b420a535-af91-4068-ae30-fef541da631b" containerName="payload-processing"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776058    6014 cpu_manager.go:414] "containerMap: removing container" podUID="6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" containerName="istio-proxy"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776062    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" containerName="istio-proxy"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776070    6014 cpu_manager.go:414] "containerMap: removing container" podUID="7c1d3787-e640-4027-b267-88a3c8a19873" containerName="payload-pre-processing"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776075    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1d3787-e640-4027-b267-88a3c8a19873" containerName="payload-pre-processing"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776084    6014 cpu_manager.go:414] "containerMap: removing container" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776088    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776094    6014 cpu_manager.go:414] "containerMap: removing container" podUID="33c08345-1df9-4df2-883d-8715226336b3" containerName="payload-processing"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776098    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c08345-1df9-4df2-883d-8715226336b3" containerName="payload-processing"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776104    6014 cpu_manager.go:414] "containerMap: removing container" podUID="a99523a9-e39a-4c4d-b06d-9e27c847ffd5" containerName="maas-api"
Oct 30 19:22:35.776163 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776108    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99523a9-e39a-4c4d-b06d-9e27c847ffd5" containerName="maas-api"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776201    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1752af1b-aa7b-4c37-8e5d-0ebc54622df0" containerName="main"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776210    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b420a535-af91-4068-ae30-fef541da631b" containerName="payload-processing"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776219    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f71e791-7dc8-41b8-aa8c-e3c4b21779cb" containerName="istio-proxy"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776225    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1d3787-e640-4027-b267-88a3c8a19873" containerName="payload-pre-processing"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776232    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="63526e12-eb15-4369-a74c-8b77f983da8d" containerName="main"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776237    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c08345-1df9-4df2-883d-8715226336b3" containerName="payload-processing"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776242    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99523a9-e39a-4c4d-b06d-9e27c847ffd5" containerName="maas-api"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776247    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="945375bc-250a-4b52-8cfc-e405096baf26" containerName="istio-proxy"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776253    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="945375bc-250a-4b52-8cfc-e405096baf26" containerName="istio-proxy"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776259    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1c8c76-4e8d-4808-a8f4-09bd5e49f831" containerName="maas-api"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776265    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c84976c-cb92-4436-99a4-d93ef5cd07d4" containerName="main"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776272    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="da64e353-542e-4e5a-8a31-ef6c1891e5f8" containerName="main"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776277    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9a80a2-6c53-40a3-a5a3-44257e705082" containerName="payload-processing"
Oct 30 19:22:35.776759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.776283    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1456cdbd-5d69-4731-9da9-6b2e803be1f9" containerName="maas-api"
Oct 30 19:22:35.781286 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.781268    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"
Oct 30 19:22:35.883629 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.883593    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:35.883820 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.883686    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:22:35.883820 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.883780    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:22:35.883820 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.883779    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:22:35.884534 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.884512    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7\" (UID: \"2d64f418-47ad-46d4-8239-4b58b25091b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"
Oct 30 19:22:35.884580 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.884544    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg6n8\" (UniqueName: \"kubernetes.io/projected/2d64f418-47ad-46d4-8239-4b58b25091b1-kube-api-access-dg6n8\") pod \"maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7\" (UID: \"2d64f418-47ad-46d4-8239-4b58b25091b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"
Oct 30 19:22:35.884739 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.884719    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7\" (UID: \"2d64f418-47ad-46d4-8239-4b58b25091b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"
Oct 30 19:22:35.885621 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.885603    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"]
Oct 30 19:22:35.986362 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.986321    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7\" (UID: \"2d64f418-47ad-46d4-8239-4b58b25091b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"
Oct 30 19:22:35.986582 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.986365    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg6n8\" (UniqueName: \"kubernetes.io/projected/2d64f418-47ad-46d4-8239-4b58b25091b1-kube-api-access-dg6n8\") pod \"maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7\" (UID: \"2d64f418-47ad-46d4-8239-4b58b25091b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"
Oct 30 19:22:35.986582 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.986544    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7\" (UID: \"2d64f418-47ad-46d4-8239-4b58b25091b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"
Oct 30 19:22:35.986993 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.986973    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7\" (UID: \"2d64f418-47ad-46d4-8239-4b58b25091b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"
Oct 30 19:22:35.989192 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:35.989168    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7\" (UID: \"2d64f418-47ad-46d4-8239-4b58b25091b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"
Oct 30 19:22:36.043804 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:36.043737    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg6n8\" (UniqueName: \"kubernetes.io/projected/2d64f418-47ad-46d4-8239-4b58b25091b1-kube-api-access-dg6n8\") pod \"maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7\" (UID: \"2d64f418-47ad-46d4-8239-4b58b25091b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"
Oct 30 19:22:36.094506 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:36.094474    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"
Oct 30 19:22:36.470035 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:36.470010    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"]
Oct 30 19:22:36.472528 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:22:36.472492    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d64f418_47ad_46d4_8239_4b58b25091b1.slice/crio-090b2e95614ff9ea41c5397f6a374ad9d1aa1b531e55e4f60a5d8593bb026316 WatchSource:0}: Error finding container 090b2e95614ff9ea41c5397f6a374ad9d1aa1b531e55e4f60a5d8593bb026316: Status 404 returned error can't find the container with id 090b2e95614ff9ea41c5397f6a374ad9d1aa1b531e55e4f60a5d8593bb026316
Oct 30 19:22:37.086164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:37.086097    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7" event={"ID":"2d64f418-47ad-46d4-8239-4b58b25091b1","Type":"ContainerStarted","Data":"9ae9230c2d401aa04fae75b68b6fdbd25a5de4e2fdac862580c8f64c036f54a2"}
Oct 30 19:22:37.086583 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:37.086177    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7" event={"ID":"2d64f418-47ad-46d4-8239-4b58b25091b1","Type":"ContainerStarted","Data":"090b2e95614ff9ea41c5397f6a374ad9d1aa1b531e55e4f60a5d8593bb026316"}
Oct 30 19:22:37.201768 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:37.201713    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7" podStartSLOduration=2.201698099 podStartE2EDuration="2.201698099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:37.200017831 +0000 UTC m=+2595.843185445" watchObservedRunningTime="2026-10-30 19:22:37.201698099 +0000 UTC m=+2595.844865713"
Oct 30 19:22:38.093015 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:38.092983    6014 generic.go:356] "Generic (PLEG): container finished" podID="2d64f418-47ad-46d4-8239-4b58b25091b1" containerID="9ae9230c2d401aa04fae75b68b6fdbd25a5de4e2fdac862580c8f64c036f54a2" exitCode=0
Oct 30 19:22:38.093480 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:38.093067    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7" event={"ID":"2d64f418-47ad-46d4-8239-4b58b25091b1","Type":"ContainerDied","Data":"9ae9230c2d401aa04fae75b68b6fdbd25a5de4e2fdac862580c8f64c036f54a2"}
Oct 30 19:22:39.239936 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:39.239905    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"
Oct 30 19:22:39.326609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:39.326564    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-openshift-ca\") pod \"2d64f418-47ad-46d4-8239-4b58b25091b1\" (UID: \"2d64f418-47ad-46d4-8239-4b58b25091b1\") "
Oct 30 19:22:39.326609 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:39.326608    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-xks-ca\") pod \"2d64f418-47ad-46d4-8239-4b58b25091b1\" (UID: \"2d64f418-47ad-46d4-8239-4b58b25091b1\") "
Oct 30 19:22:39.326880 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:39.326652    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d64f418-47ad-46d4-8239-4b58b25091b1-kube-api-access-dg6n8\" (UniqueName: \"kubernetes.io/projected/2d64f418-47ad-46d4-8239-4b58b25091b1-kube-api-access-dg6n8\") pod \"2d64f418-47ad-46d4-8239-4b58b25091b1\" (UID: \"2d64f418-47ad-46d4-8239-4b58b25091b1\") "
Oct 30 19:22:39.327034 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:39.327011    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-openshift-ca" pod "2d64f418-47ad-46d4-8239-4b58b25091b1" (UID: "2d64f418-47ad-46d4-8239-4b58b25091b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:39.328974 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:39.328941    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-xks-ca" pod "2d64f418-47ad-46d4-8239-4b58b25091b1" (UID: "2d64f418-47ad-46d4-8239-4b58b25091b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:39.329078 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:39.329014    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d64f418-47ad-46d4-8239-4b58b25091b1-kube-api-access-dg6n8" pod "2d64f418-47ad-46d4-8239-4b58b25091b1" (UID: "2d64f418-47ad-46d4-8239-4b58b25091b1"). InnerVolumeSpecName "kube-api-access-dg6n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:39.428471 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:39.428384    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg6n8\" (UniqueName: \"kubernetes.io/projected/2d64f418-47ad-46d4-8239-4b58b25091b1-kube-api-access-dg6n8\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:39.428471 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:39.428415    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:39.428471 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:39.428424    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d64f418-47ad-46d4-8239-4b58b25091b1-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:40.105045 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.105014    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"
Oct 30 19:22:40.105238 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.105049    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7" event={"ID":"2d64f418-47ad-46d4-8239-4b58b25091b1","Type":"ContainerDied","Data":"090b2e95614ff9ea41c5397f6a374ad9d1aa1b531e55e4f60a5d8593bb026316"}
Oct 30 19:22:40.105238 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.105102    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="090b2e95614ff9ea41c5397f6a374ad9d1aa1b531e55e4f60a5d8593bb026316"
Oct 30 19:22:40.146756 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.146709    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm" podUID="cedbbcda-3367-4df0-bbca-e9af8db159cd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:40.282881 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.282816    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:22:40.338590 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.338561    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-envoy\") pod \"cd2baa7c-faac-4276-b89e-2b16917d3495\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") "
Oct 30 19:22:40.338811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.338612    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-kube-api-access-749b5\" (UniqueName: \"kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-kube-api-access-749b5\") pod \"cd2baa7c-faac-4276-b89e-2b16917d3495\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") "
Oct 30 19:22:40.338811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.338661    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-socket\") pod \"cd2baa7c-faac-4276-b89e-2b16917d3495\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") "
Oct 30 19:22:40.338811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.338686    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-podinfo\") pod \"cd2baa7c-faac-4276-b89e-2b16917d3495\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") "
Oct 30 19:22:40.338811 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.338798    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-certs\") pod \"cd2baa7c-faac-4276-b89e-2b16917d3495\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") "
Oct 30 19:22:40.339048 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.338827    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-token\" (UniqueName: \"kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-token\") pod \"cd2baa7c-faac-4276-b89e-2b16917d3495\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") "
Oct 30 19:22:40.339048 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.338856    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd2baa7c-faac-4276-b89e-2b16917d3495-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2baa7c-faac-4276-b89e-2b16917d3495-istiod-ca-cert\") pod \"cd2baa7c-faac-4276-b89e-2b16917d3495\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") "
Oct 30 19:22:40.339048 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.338897    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-data\") pod \"cd2baa7c-faac-4276-b89e-2b16917d3495\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") "
Oct 30 19:22:40.339048 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.338924    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-credential-socket\") pod \"cd2baa7c-faac-4276-b89e-2b16917d3495\" (UID: \"cd2baa7c-faac-4276-b89e-2b16917d3495\") "
Oct 30 19:22:40.339048 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.338970    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-socket" pod "cd2baa7c-faac-4276-b89e-2b16917d3495" (UID: "cd2baa7c-faac-4276-b89e-2b16917d3495"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:40.339277 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.339216    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-data" pod "cd2baa7c-faac-4276-b89e-2b16917d3495" (UID: "cd2baa7c-faac-4276-b89e-2b16917d3495"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:40.339277 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.339230    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-certs" pod "cd2baa7c-faac-4276-b89e-2b16917d3495" (UID: "cd2baa7c-faac-4276-b89e-2b16917d3495"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:40.339462 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.339442    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-credential-socket" pod "cd2baa7c-faac-4276-b89e-2b16917d3495" (UID: "cd2baa7c-faac-4276-b89e-2b16917d3495"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:40.339524 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.339396    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:40.339524 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.339489    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:40.339524 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.339505    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:40.339658 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.339536    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2baa7c-faac-4276-b89e-2b16917d3495-istiod-ca-cert" pod "cd2baa7c-faac-4276-b89e-2b16917d3495" (UID: "cd2baa7c-faac-4276-b89e-2b16917d3495"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:40.341226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.341104    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-envoy" pod "cd2baa7c-faac-4276-b89e-2b16917d3495" (UID: "cd2baa7c-faac-4276-b89e-2b16917d3495"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:40.341499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.341476    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-podinfo" pod "cd2baa7c-faac-4276-b89e-2b16917d3495" (UID: "cd2baa7c-faac-4276-b89e-2b16917d3495"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:22:40.342034 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.342007    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-kube-api-access-749b5" pod "cd2baa7c-faac-4276-b89e-2b16917d3495" (UID: "cd2baa7c-faac-4276-b89e-2b16917d3495"). InnerVolumeSpecName "kube-api-access-749b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:40.342034 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.342012    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-token" pod "cd2baa7c-faac-4276-b89e-2b16917d3495" (UID: "cd2baa7c-faac-4276-b89e-2b16917d3495"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:40.440436 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.440396    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:40.440436 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.440432    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:40.440436 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.440444    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-749b5\" (UniqueName: \"kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-kube-api-access-749b5\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:40.440693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.440458    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:40.440693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.440470    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2baa7c-faac-4276-b89e-2b16917d3495-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:40.440693 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.440481    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2baa7c-faac-4276-b89e-2b16917d3495-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:40.458441 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.458399    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"]
Oct 30 19:22:40.463331 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:40.463295    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"]
Oct 30 19:22:41.111833 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:41.111801    6014 generic.go:356] "Generic (PLEG): container finished" podID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerID="d0e12575132ca27aa7dacde8999847ca451cba4646e5b1e0093b1cec125310f0" exitCode=0
Oct 30 19:22:41.112035 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:41.111855    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"
Oct 30 19:22:41.112035 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:41.111885    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9" event={"ID":"cd2baa7c-faac-4276-b89e-2b16917d3495","Type":"ContainerDied","Data":"d0e12575132ca27aa7dacde8999847ca451cba4646e5b1e0093b1cec125310f0"}
Oct 30 19:22:41.112035 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:41.111947    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9" event={"ID":"cd2baa7c-faac-4276-b89e-2b16917d3495","Type":"ContainerDied","Data":"68a1eede3e24b934927629bd4c389c758fa3719050481c61cc90f0498b31400c"}
Oct 30 19:22:41.112035 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:41.111971    6014 scope.go:122] "RemoveContainer" containerID="d0e12575132ca27aa7dacde8999847ca451cba4646e5b1e0093b1cec125310f0"
Oct 30 19:22:41.122857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:41.122697    6014 scope.go:122] "RemoveContainer" containerID="253907f23b82ed9f9d85af5cc97549680fdbaf6b8b9d5fde2abfd5bb23ff6bf7"
Oct 30 19:22:41.132022 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:41.131994    6014 scope.go:122] "RemoveContainer" containerID="d0e12575132ca27aa7dacde8999847ca451cba4646e5b1e0093b1cec125310f0"
Oct 30 19:22:41.132349 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:22:41.132331    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e12575132ca27aa7dacde8999847ca451cba4646e5b1e0093b1cec125310f0\": container with ID starting with d0e12575132ca27aa7dacde8999847ca451cba4646e5b1e0093b1cec125310f0 not found: ID does not exist" containerID="d0e12575132ca27aa7dacde8999847ca451cba4646e5b1e0093b1cec125310f0"
Oct 30 19:22:41.132433 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:41.132363    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e12575132ca27aa7dacde8999847ca451cba4646e5b1e0093b1cec125310f0"} err="failed to get container status \"d0e12575132ca27aa7dacde8999847ca451cba4646e5b1e0093b1cec125310f0\": rpc error: code = NotFound desc = could not find container \"d0e12575132ca27aa7dacde8999847ca451cba4646e5b1e0093b1cec125310f0\": container with ID starting with d0e12575132ca27aa7dacde8999847ca451cba4646e5b1e0093b1cec125310f0 not found: ID does not exist"
Oct 30 19:22:41.132433 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:41.132388    6014 scope.go:122] "RemoveContainer" containerID="253907f23b82ed9f9d85af5cc97549680fdbaf6b8b9d5fde2abfd5bb23ff6bf7"
Oct 30 19:22:41.132654 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:22:41.132636    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253907f23b82ed9f9d85af5cc97549680fdbaf6b8b9d5fde2abfd5bb23ff6bf7\": container with ID starting with 253907f23b82ed9f9d85af5cc97549680fdbaf6b8b9d5fde2abfd5bb23ff6bf7 not found: ID does not exist" containerID="253907f23b82ed9f9d85af5cc97549680fdbaf6b8b9d5fde2abfd5bb23ff6bf7"
Oct 30 19:22:41.132717 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:41.132663    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253907f23b82ed9f9d85af5cc97549680fdbaf6b8b9d5fde2abfd5bb23ff6bf7"} err="failed to get container status \"253907f23b82ed9f9d85af5cc97549680fdbaf6b8b9d5fde2abfd5bb23ff6bf7\": rpc error: code = NotFound desc = could not find container \"253907f23b82ed9f9d85af5cc97549680fdbaf6b8b9d5fde2abfd5bb23ff6bf7\": container with ID starting with 253907f23b82ed9f9d85af5cc97549680fdbaf6b8b9d5fde2abfd5bb23ff6bf7 not found: ID does not exist"
Oct 30 19:22:41.275312 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:41.275275    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"]
Oct 30 19:22:41.324995 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:41.324959    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-714b2c-openshift-default-5f7c448f9-8xrp9"]
Oct 30 19:22:41.482014 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:41.481929    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd2baa7c-faac-4276-b89e-2b16917d3495" path="/var/lib/kubelet/pods/cd2baa7c-faac-4276-b89e-2b16917d3495/volumes"
Oct 30 19:22:44.917900 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:44.917875    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"
Oct 30 19:22:44.997751 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:44.997659    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cedbbcda-3367-4df0-bbca-e9af8db159cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cedbbcda-3367-4df0-bbca-e9af8db159cd-plugins-config-volume\") pod \"cedbbcda-3367-4df0-bbca-e9af8db159cd\" (UID: \"cedbbcda-3367-4df0-bbca-e9af8db159cd\") "
Oct 30 19:22:44.997944 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:44.997824    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cedbbcda-3367-4df0-bbca-e9af8db159cd-kube-api-access-cp65l\" (UniqueName: \"kubernetes.io/projected/cedbbcda-3367-4df0-bbca-e9af8db159cd-kube-api-access-cp65l\") pod \"cedbbcda-3367-4df0-bbca-e9af8db159cd\" (UID: \"cedbbcda-3367-4df0-bbca-e9af8db159cd\") "
Oct 30 19:22:44.998120 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:44.998090    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cedbbcda-3367-4df0-bbca-e9af8db159cd-plugins-config-volume" pod "cedbbcda-3367-4df0-bbca-e9af8db159cd" (UID: "cedbbcda-3367-4df0-bbca-e9af8db159cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:45.000087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:45.000065    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cedbbcda-3367-4df0-bbca-e9af8db159cd-kube-api-access-cp65l" pod "cedbbcda-3367-4df0-bbca-e9af8db159cd" (UID: "cedbbcda-3367-4df0-bbca-e9af8db159cd"). InnerVolumeSpecName "kube-api-access-cp65l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:45.099636 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:45.099595    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cedbbcda-3367-4df0-bbca-e9af8db159cd-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.099636 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:45.099628    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp65l\" (UniqueName: \"kubernetes.io/projected/cedbbcda-3367-4df0-bbca-e9af8db159cd-kube-api-access-cp65l\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.134238 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:45.134204    6014 generic.go:356] "Generic (PLEG): container finished" podID="cedbbcda-3367-4df0-bbca-e9af8db159cd" containerID="0bc8a1e0da64b982aa04f7457f1ce49cc2b7b44b853e800c15328519c93f01de" exitCode=0
Oct 30 19:22:45.134386 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:45.134258    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"
Oct 30 19:22:45.134386 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:45.134295    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm" event={"ID":"cedbbcda-3367-4df0-bbca-e9af8db159cd","Type":"ContainerDied","Data":"0bc8a1e0da64b982aa04f7457f1ce49cc2b7b44b853e800c15328519c93f01de"}
Oct 30 19:22:45.134386 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:45.134356    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm" event={"ID":"cedbbcda-3367-4df0-bbca-e9af8db159cd","Type":"ContainerDied","Data":"c07957956c8549a285f33dca0a6711cc123093e8be507b33405edb689f84b099"}
Oct 30 19:22:45.134386 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:45.134384    6014 scope.go:122] "RemoveContainer" containerID="0bc8a1e0da64b982aa04f7457f1ce49cc2b7b44b853e800c15328519c93f01de"
Oct 30 19:22:45.144041 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:45.144008    6014 scope.go:122] "RemoveContainer" containerID="0bc8a1e0da64b982aa04f7457f1ce49cc2b7b44b853e800c15328519c93f01de"
Oct 30 19:22:45.144357 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:22:45.144337    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc8a1e0da64b982aa04f7457f1ce49cc2b7b44b853e800c15328519c93f01de\": container with ID starting with 0bc8a1e0da64b982aa04f7457f1ce49cc2b7b44b853e800c15328519c93f01de not found: ID does not exist" containerID="0bc8a1e0da64b982aa04f7457f1ce49cc2b7b44b853e800c15328519c93f01de"
Oct 30 19:22:45.144448 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:45.144365    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc8a1e0da64b982aa04f7457f1ce49cc2b7b44b853e800c15328519c93f01de"} err="failed to get container status \"0bc8a1e0da64b982aa04f7457f1ce49cc2b7b44b853e800c15328519c93f01de\": rpc error: code = NotFound desc = could not find container \"0bc8a1e0da64b982aa04f7457f1ce49cc2b7b44b853e800c15328519c93f01de\": container with ID starting with 0bc8a1e0da64b982aa04f7457f1ce49cc2b7b44b853e800c15328519c93f01de not found: ID does not exist"
Oct 30 19:22:45.296779 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:45.296689    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"]
Oct 30 19:22:45.300670 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:45.300644    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-714b2c-59d9d66c79-kd5tm"]
Oct 30 19:22:45.475345 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:45.475318    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cedbbcda-3367-4df0-bbca-e9af8db159cd" path="/var/lib/kubelet/pods/cedbbcda-3367-4df0-bbca-e9af8db159cd/volumes"
Oct 30 19:22:47.200266 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:47.200226    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"]
Oct 30 19:22:50.774508 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.774468    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"]
Oct 30 19:22:50.775057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.774949    6014 cpu_manager.go:414] "containerMap: removing container" podUID="cedbbcda-3367-4df0-bbca-e9af8db159cd" containerName="payload-pre-processing"
Oct 30 19:22:50.775057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.774961    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cedbbcda-3367-4df0-bbca-e9af8db159cd" containerName="payload-pre-processing"
Oct 30 19:22:50.775057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.774976    6014 cpu_manager.go:414] "containerMap: removing container" podUID="2d64f418-47ad-46d4-8239-4b58b25091b1" containerName="revoke-keys"
Oct 30 19:22:50.775057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.774981    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d64f418-47ad-46d4-8239-4b58b25091b1" containerName="revoke-keys"
Oct 30 19:22:50.775057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.774996    6014 cpu_manager.go:414] "containerMap: removing container" podUID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerName="istio-proxy"
Oct 30 19:22:50.775057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.775001    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerName="istio-proxy"
Oct 30 19:22:50.775057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.775010    6014 cpu_manager.go:414] "containerMap: removing container" podUID="945375bc-250a-4b52-8cfc-e405096baf26" containerName="istio-proxy"
Oct 30 19:22:50.775057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.775014    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945375bc-250a-4b52-8cfc-e405096baf26" containerName="istio-proxy"
Oct 30 19:22:50.775057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.775021    6014 cpu_manager.go:414] "containerMap: removing container" podUID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerName="istio-proxy"
Oct 30 19:22:50.775057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.775025    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerName="istio-proxy"
Oct 30 19:22:50.775575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.775092    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="cedbbcda-3367-4df0-bbca-e9af8db159cd" containerName="payload-pre-processing"
Oct 30 19:22:50.775575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.775099    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d64f418-47ad-46d4-8239-4b58b25091b1" containerName="revoke-keys"
Oct 30 19:22:50.775575 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.775108    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerName="istio-proxy"
Oct 30 19:22:50.778128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.778110    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"
Oct 30 19:22:50.829879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.829851    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:22:50.868419 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.868386    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a13d524a-lrcdm\" (UID: \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"
Oct 30 19:22:50.868613 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.868443    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzpcv\" (UniqueName: \"kubernetes.io/projected/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-kube-api-access-tzpcv\") pod \"maas-api-revoke-sub-a13d524a-lrcdm\" (UID: \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"
Oct 30 19:22:50.868697 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.868666    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a13d524a-lrcdm\" (UID: \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"
Oct 30 19:22:50.879373 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.879344    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:50.879516 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.879347    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:22:50.879516 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.879386    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:22:50.881329 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.881307    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"]
Oct 30 19:22:50.969908 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.969874    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a13d524a-lrcdm\" (UID: \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"
Oct 30 19:22:50.970107 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.969927    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a13d524a-lrcdm\" (UID: \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"
Oct 30 19:22:50.970107 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.969962    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzpcv\" (UniqueName: \"kubernetes.io/projected/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-kube-api-access-tzpcv\") pod \"maas-api-revoke-sub-a13d524a-lrcdm\" (UID: \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"
Oct 30 19:22:50.970706 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.970677    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a13d524a-lrcdm\" (UID: \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"
Oct 30 19:22:50.972515 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:50.972497    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a13d524a-lrcdm\" (UID: \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"
Oct 30 19:22:51.028105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:51.028035    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzpcv\" (UniqueName: \"kubernetes.io/projected/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-kube-api-access-tzpcv\") pod \"maas-api-revoke-sub-a13d524a-lrcdm\" (UID: \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"
Oct 30 19:22:51.090927 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:51.090893    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"
Oct 30 19:22:51.465172 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:51.465119    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"]
Oct 30 19:22:51.468427 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:22:51.468389    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d89e2e9_48ab_44c2_9fd4_6c616edb400e.slice/crio-641b6a29f37a77e8274772cc4a9983248d990b0b0fc1629c454ed9c4a42e97eb WatchSource:0}: Error finding container 641b6a29f37a77e8274772cc4a9983248d990b0b0fc1629c454ed9c4a42e97eb: Status 404 returned error can't find the container with id 641b6a29f37a77e8274772cc4a9983248d990b0b0fc1629c454ed9c4a42e97eb
Oct 30 19:22:52.173892 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:52.173853    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm" event={"ID":"9d89e2e9-48ab-44c2-9fd4-6c616edb400e","Type":"ContainerStarted","Data":"36c4f72d57c4a7a90e9e7eff7ac41a1768d205f5298d9e72adc88bc60976f561"}
Oct 30 19:22:52.174377 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:52.173910    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm" event={"ID":"9d89e2e9-48ab-44c2-9fd4-6c616edb400e","Type":"ContainerStarted","Data":"641b6a29f37a77e8274772cc4a9983248d990b0b0fc1629c454ed9c4a42e97eb"}
Oct 30 19:22:52.201321 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:52.201283    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t" podUID="ec095b92-3649-4328-b5c7-fb47d3d5a9ae" containerName="payload-processing" containerID="cri-o://85bab69eb18b667a328cced4bc5f55092f03a5f8923e09a13cc635e65e455700" gracePeriod=25
Oct 30 19:22:52.316830 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:52.316770    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm" podStartSLOduration=2.316752804 podStartE2EDuration="2.316752804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:52.314479023 +0000 UTC m=+2610.957646637" watchObservedRunningTime="2026-10-30 19:22:52.316752804 +0000 UTC m=+2610.959920415"
Oct 30 19:22:53.181601 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:53.181564    6014 generic.go:356] "Generic (PLEG): container finished" podID="9d89e2e9-48ab-44c2-9fd4-6c616edb400e" containerID="36c4f72d57c4a7a90e9e7eff7ac41a1768d205f5298d9e72adc88bc60976f561" exitCode=0
Oct 30 19:22:53.182009 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:53.181633    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm" event={"ID":"9d89e2e9-48ab-44c2-9fd4-6c616edb400e","Type":"ContainerDied","Data":"36c4f72d57c4a7a90e9e7eff7ac41a1768d205f5298d9e72adc88bc60976f561"}
Oct 30 19:22:53.257783 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:22:53.257742    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:22:54.330189 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:54.330162    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"
Oct 30 19:22:54.508302 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:54.508210    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-xks-ca\") pod \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\" (UID: \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\") "
Oct 30 19:22:54.508480 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:54.508357    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-kube-api-access-tzpcv\" (UniqueName: \"kubernetes.io/projected/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-kube-api-access-tzpcv\") pod \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\" (UID: \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\") "
Oct 30 19:22:54.508480 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:54.508385    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-openshift-ca\") pod \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\" (UID: \"9d89e2e9-48ab-44c2-9fd4-6c616edb400e\") "
Oct 30 19:22:54.508776 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:54.508753    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-openshift-ca" pod "9d89e2e9-48ab-44c2-9fd4-6c616edb400e" (UID: "9d89e2e9-48ab-44c2-9fd4-6c616edb400e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:54.510413 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:54.510390    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-xks-ca" pod "9d89e2e9-48ab-44c2-9fd4-6c616edb400e" (UID: "9d89e2e9-48ab-44c2-9fd4-6c616edb400e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:54.510636 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:54.510619    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-kube-api-access-tzpcv" pod "9d89e2e9-48ab-44c2-9fd4-6c616edb400e" (UID: "9d89e2e9-48ab-44c2-9fd4-6c616edb400e"). InnerVolumeSpecName "kube-api-access-tzpcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:54.610400 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:54.610356    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-xks-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:54.610400 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:54.610398    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzpcv\" (UniqueName: \"kubernetes.io/projected/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-kube-api-access-tzpcv\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:54.610671 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:54.610414    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d89e2e9-48ab-44c2-9fd4-6c616edb400e-maas-api-openshift-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:55.193849 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.193814    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm" event={"ID":"9d89e2e9-48ab-44c2-9fd4-6c616edb400e","Type":"ContainerDied","Data":"641b6a29f37a77e8274772cc4a9983248d990b0b0fc1629c454ed9c4a42e97eb"}
Oct 30 19:22:55.193849 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.193845    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"
Oct 30 19:22:55.194085 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.193873    6014 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="641b6a29f37a77e8274772cc4a9983248d990b0b0fc1629c454ed9c4a42e97eb"
Oct 30 19:22:55.480265 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.480180    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" containerID="cri-o://2a644e0575eabebac5a50dcf71e57d9bf02f21b07f098456f255896862c10409" gracePeriod=45
Oct 30 19:22:55.483815 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.483785    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" containerID="cri-o://174b6520e0e48214bf8fcf5514bc093982c7ead3368ca3beb042f0c045dd2e0d" gracePeriod=45
Oct 30 19:22:55.617039 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.616992    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main" probeResult="failure" output="Get \"https://10.134.0.105:8000/ready\": dial tcp 10.134.0.105:8000: connect: connection refused"
Oct 30 19:22:55.673457 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.673212    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 19:22:55.739899 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.739835    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:22:55.765577 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.765546    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:22:55.924933 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.924901    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-model-cache\") pod \"355c74b1-f4af-41d2-a557-93b3e62f7499\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") "
Oct 30 19:22:55.926128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.924947    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/355c74b1-f4af-41d2-a557-93b3e62f7499-tls-certs\" (UniqueName: \"kubernetes.io/secret/355c74b1-f4af-41d2-a557-93b3e62f7499-tls-certs\") pod \"355c74b1-f4af-41d2-a557-93b3e62f7499\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") "
Oct 30 19:22:55.926128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.924983    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-model-cache\") pod \"9b7c2796-648a-48b1-b6a7-d83661404040\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") "
Oct 30 19:22:55.926128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.925024    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-dshm\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-dshm\") pod \"355c74b1-f4af-41d2-a557-93b3e62f7499\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") "
Oct 30 19:22:55.926128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.925089    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-tmp-dir\") pod \"355c74b1-f4af-41d2-a557-93b3e62f7499\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") "
Oct 30 19:22:55.926128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.925322    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-dshm\") pod \"9b7c2796-648a-48b1-b6a7-d83661404040\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") "
Oct 30 19:22:55.926128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.925390    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-home\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-home\") pod \"355c74b1-f4af-41d2-a557-93b3e62f7499\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") "
Oct 30 19:22:55.926128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.925439    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355c74b1-f4af-41d2-a557-93b3e62f7499-kube-api-access-9c7x7\" (UniqueName: \"kubernetes.io/projected/355c74b1-f4af-41d2-a557-93b3e62f7499-kube-api-access-9c7x7\") pod \"355c74b1-f4af-41d2-a557-93b3e62f7499\" (UID: \"355c74b1-f4af-41d2-a557-93b3e62f7499\") "
Oct 30 19:22:55.926128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.925475    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7c2796-648a-48b1-b6a7-d83661404040-kube-api-access-7pbcz\" (UniqueName: \"kubernetes.io/projected/9b7c2796-648a-48b1-b6a7-d83661404040-kube-api-access-7pbcz\") pod \"9b7c2796-648a-48b1-b6a7-d83661404040\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") "
Oct 30 19:22:55.926128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.925519    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-tmp-dir\") pod \"9b7c2796-648a-48b1-b6a7-d83661404040\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") "
Oct 30 19:22:55.926128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.925595    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-home\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-home\") pod \"9b7c2796-648a-48b1-b6a7-d83661404040\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") "
Oct 30 19:22:55.926128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.925739    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b7c2796-648a-48b1-b6a7-d83661404040-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7c2796-648a-48b1-b6a7-d83661404040-tls-certs\") pod \"9b7c2796-648a-48b1-b6a7-d83661404040\" (UID: \"9b7c2796-648a-48b1-b6a7-d83661404040\") "
Oct 30 19:22:55.927719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.926323    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-tmp-dir" pod "355c74b1-f4af-41d2-a557-93b3e62f7499" (UID: "355c74b1-f4af-41d2-a557-93b3e62f7499"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:55.927719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.926773    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-model-cache" pod "9b7c2796-648a-48b1-b6a7-d83661404040" (UID: "9b7c2796-648a-48b1-b6a7-d83661404040"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:55.927719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.926782    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-home" pod "355c74b1-f4af-41d2-a557-93b3e62f7499" (UID: "355c74b1-f4af-41d2-a557-93b3e62f7499"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:55.927719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.927189    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-model-cache" pod "355c74b1-f4af-41d2-a557-93b3e62f7499" (UID: "355c74b1-f4af-41d2-a557-93b3e62f7499"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:55.927719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.927475    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-home" pod "9b7c2796-648a-48b1-b6a7-d83661404040" (UID: "9b7c2796-648a-48b1-b6a7-d83661404040"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:55.928022 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.927731    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-tmp-dir" pod "9b7c2796-648a-48b1-b6a7-d83661404040" (UID: "9b7c2796-648a-48b1-b6a7-d83661404040"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:55.929024 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.928995    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355c74b1-f4af-41d2-a557-93b3e62f7499-tls-certs" pod "355c74b1-f4af-41d2-a557-93b3e62f7499" (UID: "355c74b1-f4af-41d2-a557-93b3e62f7499"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:55.929200 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.929159    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7c2796-648a-48b1-b6a7-d83661404040-tls-certs" pod "9b7c2796-648a-48b1-b6a7-d83661404040" (UID: "9b7c2796-648a-48b1-b6a7-d83661404040"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:55.930375 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.930354    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-dshm" pod "9b7c2796-648a-48b1-b6a7-d83661404040" (UID: "9b7c2796-648a-48b1-b6a7-d83661404040"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:55.930451 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.930409    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-dshm" pod "355c74b1-f4af-41d2-a557-93b3e62f7499" (UID: "355c74b1-f4af-41d2-a557-93b3e62f7499"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:55.930451 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.930417    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355c74b1-f4af-41d2-a557-93b3e62f7499-kube-api-access-9c7x7" pod "355c74b1-f4af-41d2-a557-93b3e62f7499" (UID: "355c74b1-f4af-41d2-a557-93b3e62f7499"). InnerVolumeSpecName "kube-api-access-9c7x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:55.930888 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:55.930860    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7c2796-648a-48b1-b6a7-d83661404040-kube-api-access-7pbcz" pod "9b7c2796-648a-48b1-b6a7-d83661404040" (UID: "9b7c2796-648a-48b1-b6a7-d83661404040"). InnerVolumeSpecName "kube-api-access-7pbcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:56.027193 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.027078    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c7x7\" (UniqueName: \"kubernetes.io/projected/355c74b1-f4af-41d2-a557-93b3e62f7499-kube-api-access-9c7x7\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:56.027193 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.027107    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pbcz\" (UniqueName: \"kubernetes.io/projected/9b7c2796-648a-48b1-b6a7-d83661404040-kube-api-access-7pbcz\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:56.027193 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.027120    6014 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-tmp-dir\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:56.027193 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.027128    6014 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-home\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:56.027193 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.027156    6014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7c2796-648a-48b1-b6a7-d83661404040-tls-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:56.027193 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.027169    6014 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-model-cache\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:56.027193 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.027177    6014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355c74b1-f4af-41d2-a557-93b3e62f7499-tls-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:56.027193 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.027189    6014 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-model-cache\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:56.027193 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.027197    6014 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-dshm\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:56.027677 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.027205    6014 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-tmp-dir\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:56.027677 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.027212    6014 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7c2796-648a-48b1-b6a7-d83661404040-dshm\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:56.027677 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.027219    6014 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/355c74b1-f4af-41d2-a557-93b3e62f7499-home\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:56.200988 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.200957    6014 generic.go:356] "Generic (PLEG): container finished" podID="9b7c2796-648a-48b1-b6a7-d83661404040" containerID="2a644e0575eabebac5a50dcf71e57d9bf02f21b07f098456f255896862c10409" exitCode=0
Oct 30 19:22:56.201209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.201001    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"
Oct 30 19:22:56.201209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.201041    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" event={"ID":"9b7c2796-648a-48b1-b6a7-d83661404040","Type":"ContainerDied","Data":"2a644e0575eabebac5a50dcf71e57d9bf02f21b07f098456f255896862c10409"}
Oct 30 19:22:56.201209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.201087    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6" event={"ID":"9b7c2796-648a-48b1-b6a7-d83661404040","Type":"ContainerDied","Data":"f3e92e2e8ce604cfb325237cb7337151671d2971a130e0b0db90ae8eec8b813f"}
Oct 30 19:22:56.201209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.201115    6014 scope.go:122] "RemoveContainer" containerID="2a644e0575eabebac5a50dcf71e57d9bf02f21b07f098456f255896862c10409"
Oct 30 19:22:56.202591 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.202533    6014 generic.go:356] "Generic (PLEG): container finished" podID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerID="174b6520e0e48214bf8fcf5514bc093982c7ead3368ca3beb042f0c045dd2e0d" exitCode=0
Oct 30 19:22:56.202591 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.202583    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"
Oct 30 19:22:56.202950 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.202575    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" event={"ID":"355c74b1-f4af-41d2-a557-93b3e62f7499","Type":"ContainerDied","Data":"174b6520e0e48214bf8fcf5514bc093982c7ead3368ca3beb042f0c045dd2e0d"}
Oct 30 19:22:56.202950 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.202694    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx" event={"ID":"355c74b1-f4af-41d2-a557-93b3e62f7499","Type":"ContainerDied","Data":"29e82832f224ccff3e32b3867442acc960a8a88ea07842766a8e26f3826f2491"}
Oct 30 19:22:56.210703 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.210684    6014 scope.go:122] "RemoveContainer" containerID="2a644e0575eabebac5a50dcf71e57d9bf02f21b07f098456f255896862c10409"
Oct 30 19:22:56.211036 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:22:56.211011    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a644e0575eabebac5a50dcf71e57d9bf02f21b07f098456f255896862c10409\": container with ID starting with 2a644e0575eabebac5a50dcf71e57d9bf02f21b07f098456f255896862c10409 not found: ID does not exist" containerID="2a644e0575eabebac5a50dcf71e57d9bf02f21b07f098456f255896862c10409"
Oct 30 19:22:56.211175 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.211045    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a644e0575eabebac5a50dcf71e57d9bf02f21b07f098456f255896862c10409"} err="failed to get container status \"2a644e0575eabebac5a50dcf71e57d9bf02f21b07f098456f255896862c10409\": rpc error: code = NotFound desc = could not find container \"2a644e0575eabebac5a50dcf71e57d9bf02f21b07f098456f255896862c10409\": container with ID starting with 2a644e0575eabebac5a50dcf71e57d9bf02f21b07f098456f255896862c10409 not found: ID does not exist"
Oct 30 19:22:56.211175 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.211075    6014 scope.go:122] "RemoveContainer" containerID="174b6520e0e48214bf8fcf5514bc093982c7ead3368ca3beb042f0c045dd2e0d"
Oct 30 19:22:56.219505 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.219486    6014 scope.go:122] "RemoveContainer" containerID="174b6520e0e48214bf8fcf5514bc093982c7ead3368ca3beb042f0c045dd2e0d"
Oct 30 19:22:56.219764 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:22:56.219747    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174b6520e0e48214bf8fcf5514bc093982c7ead3368ca3beb042f0c045dd2e0d\": container with ID starting with 174b6520e0e48214bf8fcf5514bc093982c7ead3368ca3beb042f0c045dd2e0d not found: ID does not exist" containerID="174b6520e0e48214bf8fcf5514bc093982c7ead3368ca3beb042f0c045dd2e0d"
Oct 30 19:22:56.219815 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.219773    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174b6520e0e48214bf8fcf5514bc093982c7ead3368ca3beb042f0c045dd2e0d"} err="failed to get container status \"174b6520e0e48214bf8fcf5514bc093982c7ead3368ca3beb042f0c045dd2e0d\": rpc error: code = NotFound desc = could not find container \"174b6520e0e48214bf8fcf5514bc093982c7ead3368ca3beb042f0c045dd2e0d\": container with ID starting with 174b6520e0e48214bf8fcf5514bc093982c7ead3368ca3beb042f0c045dd2e0d not found: ID does not exist"
Oct 30 19:22:56.364485 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.364444    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"]
Oct 30 19:22:56.413831 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.413797    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/distinct-2-w2-66bdc0-kserve-7755566b57-k4hn6"]
Oct 30 19:22:56.522773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.522739    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"]
Oct 30 19:22:56.527728 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:56.527703    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/distinct-w2-66bdc0-kserve-7f89597b4b-p28qx"]
Oct 30 19:22:57.475017 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:57.474980    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" path="/var/lib/kubelet/pods/355c74b1-f4af-41d2-a557-93b3e62f7499/volumes"
Oct 30 19:22:57.475456 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:57.475440    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" path="/var/lib/kubelet/pods/9b7c2796-648a-48b1-b6a7-d83661404040/volumes"
Oct 30 19:22:58.124759 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:22:58.124721    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t" podUID="ec095b92-3649-4328-b5c7-fb47d3d5a9ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:05.962993 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:05.962898    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:23:08.124692 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:08.124654    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t" podUID="ec095b92-3649-4328-b5c7-fb47d3d5a9ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:08.215656 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:08.215622    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"]
Oct 30 19:23:08.229830 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:08.229803    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-66bdc0-zpmz7"]
Oct 30 19:23:09.474735 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:09.474701    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d64f418-47ad-46d4-8239-4b58b25091b1" path="/var/lib/kubelet/pods/2d64f418-47ad-46d4-8239-4b58b25091b1/volumes"
Oct 30 19:23:12.458059 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:12.457991    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"
Oct 30 19:23:12.492859 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:12.492824    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-kube-api-access-hvkdb\" (UniqueName: \"kubernetes.io/projected/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-kube-api-access-hvkdb\") pod \"ec095b92-3649-4328-b5c7-fb47d3d5a9ae\" (UID: \"ec095b92-3649-4328-b5c7-fb47d3d5a9ae\") "
Oct 30 19:23:12.493030 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:12.492941    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-plugins-config-volume\") pod \"ec095b92-3649-4328-b5c7-fb47d3d5a9ae\" (UID: \"ec095b92-3649-4328-b5c7-fb47d3d5a9ae\") "
Oct 30 19:23:12.493354 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:12.493323    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-plugins-config-volume" pod "ec095b92-3649-4328-b5c7-fb47d3d5a9ae" (UID: "ec095b92-3649-4328-b5c7-fb47d3d5a9ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:12.493448 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:12.493419    6014 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-plugins-config-volume\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:12.495417 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:12.495382    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-kube-api-access-hvkdb" pod "ec095b92-3649-4328-b5c7-fb47d3d5a9ae" (UID: "ec095b92-3649-4328-b5c7-fb47d3d5a9ae"). InnerVolumeSpecName "kube-api-access-hvkdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:12.594950 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:12.594903    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvkdb\" (UniqueName: \"kubernetes.io/projected/ec095b92-3649-4328-b5c7-fb47d3d5a9ae-kube-api-access-hvkdb\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:13.311303 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:13.311269    6014 generic.go:356] "Generic (PLEG): container finished" podID="ec095b92-3649-4328-b5c7-fb47d3d5a9ae" containerID="85bab69eb18b667a328cced4bc5f55092f03a5f8923e09a13cc635e65e455700" exitCode=0
Oct 30 19:23:13.311524 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:13.311320    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"
Oct 30 19:23:13.311524 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:13.311355    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t" event={"ID":"ec095b92-3649-4328-b5c7-fb47d3d5a9ae","Type":"ContainerDied","Data":"85bab69eb18b667a328cced4bc5f55092f03a5f8923e09a13cc635e65e455700"}
Oct 30 19:23:13.311524 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:13.311425    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t" event={"ID":"ec095b92-3649-4328-b5c7-fb47d3d5a9ae","Type":"ContainerDied","Data":"7ac0f95d713aeb16d0d925fff6c0402f0b23c331f1d3cc066b394b113101dab4"}
Oct 30 19:23:13.311524 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:13.311454    6014 scope.go:122] "RemoveContainer" containerID="85bab69eb18b667a328cced4bc5f55092f03a5f8923e09a13cc635e65e455700"
Oct 30 19:23:13.321388 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:13.321209    6014 scope.go:122] "RemoveContainer" containerID="85bab69eb18b667a328cced4bc5f55092f03a5f8923e09a13cc635e65e455700"
Oct 30 19:23:13.321573 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:23:13.321550    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85bab69eb18b667a328cced4bc5f55092f03a5f8923e09a13cc635e65e455700\": container with ID starting with 85bab69eb18b667a328cced4bc5f55092f03a5f8923e09a13cc635e65e455700 not found: ID does not exist" containerID="85bab69eb18b667a328cced4bc5f55092f03a5f8923e09a13cc635e65e455700"
Oct 30 19:23:13.321620 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:13.321589    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85bab69eb18b667a328cced4bc5f55092f03a5f8923e09a13cc635e65e455700"} err="failed to get container status \"85bab69eb18b667a328cced4bc5f55092f03a5f8923e09a13cc635e65e455700\": rpc error: code = NotFound desc = could not find container \"85bab69eb18b667a328cced4bc5f55092f03a5f8923e09a13cc635e65e455700\": container with ID starting with 85bab69eb18b667a328cced4bc5f55092f03a5f8923e09a13cc635e65e455700 not found: ID does not exist"
Oct 30 19:23:13.475711 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:13.475676    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"]
Oct 30 19:23:13.482450 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:13.482424    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-66bdc0-58c9b9dd9-tdm7t"]
Oct 30 19:23:15.475024 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:15.474988    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec095b92-3649-4328-b5c7-fb47d3d5a9ae" path="/var/lib/kubelet/pods/ec095b92-3649-4328-b5c7-fb47d3d5a9ae/volumes"
Oct 30 19:23:22.723614 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.723582    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:23:22.870500 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.870466    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-994975fdf-wp825"]
Oct 30 19:23:22.870922 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.870910    6014 cpu_manager.go:414] "containerMap: removing container" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main"
Oct 30 19:23:22.870966 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.870924    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main"
Oct 30 19:23:22.870966 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.870942    6014 cpu_manager.go:414] "containerMap: removing container" podUID="9d89e2e9-48ab-44c2-9fd4-6c616edb400e" containerName="revoke-keys"
Oct 30 19:23:22.870966 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.870947    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d89e2e9-48ab-44c2-9fd4-6c616edb400e" containerName="revoke-keys"
Oct 30 19:23:22.871072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.870970    6014 cpu_manager.go:414] "containerMap: removing container" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main"
Oct 30 19:23:22.871072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.870975    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main"
Oct 30 19:23:22.871072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.870988    6014 cpu_manager.go:414] "containerMap: removing container" podUID="ec095b92-3649-4328-b5c7-fb47d3d5a9ae" containerName="payload-processing"
Oct 30 19:23:22.871072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.870993    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec095b92-3649-4328-b5c7-fb47d3d5a9ae" containerName="payload-processing"
Oct 30 19:23:22.871072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.871049    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd2baa7c-faac-4276-b89e-2b16917d3495" containerName="istio-proxy"
Oct 30 19:23:22.871072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.871061    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="355c74b1-f4af-41d2-a557-93b3e62f7499" containerName="main"
Oct 30 19:23:22.871072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.871070    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec095b92-3649-4328-b5c7-fb47d3d5a9ae" containerName="payload-processing"
Oct 30 19:23:22.871481 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.871077    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7c2796-648a-48b1-b6a7-d83661404040" containerName="main"
Oct 30 19:23:22.871481 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.871086    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d89e2e9-48ab-44c2-9fd4-6c616edb400e" containerName="revoke-keys"
Oct 30 19:23:22.874494 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.874475    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:22.926605 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.926523    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 19:23:22.976228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.976185    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 19:23:22.976228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.976215    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 19:23:22.976228 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.976186    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wwn5x\""
Oct 30 19:23:22.978466 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:22.978433    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-994975fdf-wp825"]
Oct 30 19:23:23.002799 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.002762    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmb8b\" (UniqueName: \"kubernetes.io/projected/e7961869-97b8-43e8-8a90-9c0cc539dc44-kube-api-access-rmb8b\") pod \"maas-controller-994975fdf-wp825\" (UID: \"e7961869-97b8-43e8-8a90-9c0cc539dc44\") " pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:23.002799 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.002818    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-webhook-cert\") pod \"maas-controller-994975fdf-wp825\" (UID: \"e7961869-97b8-43e8-8a90-9c0cc539dc44\") " pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:23.003072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.002939    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-metrics-certs\") pod \"maas-controller-994975fdf-wp825\" (UID: \"e7961869-97b8-43e8-8a90-9c0cc539dc44\") " pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:23.058247 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.058212    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"]
Oct 30 19:23:23.065941 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.065816    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86875e66-dcrn4"]
Oct 30 19:23:23.104572 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.104513    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmb8b\" (UniqueName: \"kubernetes.io/projected/e7961869-97b8-43e8-8a90-9c0cc539dc44-kube-api-access-rmb8b\") pod \"maas-controller-994975fdf-wp825\" (UID: \"e7961869-97b8-43e8-8a90-9c0cc539dc44\") " pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:23.104572 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.104575    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-webhook-cert\") pod \"maas-controller-994975fdf-wp825\" (UID: \"e7961869-97b8-43e8-8a90-9c0cc539dc44\") " pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:23.104914 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.104637    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-metrics-certs\") pod \"maas-controller-994975fdf-wp825\" (UID: \"e7961869-97b8-43e8-8a90-9c0cc539dc44\") " pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:23.107487 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.107457    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-webhook-cert\") pod \"maas-controller-994975fdf-wp825\" (UID: \"e7961869-97b8-43e8-8a90-9c0cc539dc44\") " pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:23.107832 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.107813    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-metrics-certs\") pod \"maas-controller-994975fdf-wp825\" (UID: \"e7961869-97b8-43e8-8a90-9c0cc539dc44\") " pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:23.162156 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.162102    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmb8b\" (UniqueName: \"kubernetes.io/projected/e7961869-97b8-43e8-8a90-9c0cc539dc44-kube-api-access-rmb8b\") pod \"maas-controller-994975fdf-wp825\" (UID: \"e7961869-97b8-43e8-8a90-9c0cc539dc44\") " pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:23.189622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.189470    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:23.265747 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:23:23.264165    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:23:23.477918 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.477826    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e84da3-cc5f-493f-8aa5-2d566333062f" path="/var/lib/kubelet/pods/e0e84da3-cc5f-493f-8aa5-2d566333062f/volumes"
Oct 30 19:23:23.580777 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.580744    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-994975fdf-wp825"]
Oct 30 19:23:23.586886 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:23:23.586850    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7961869_97b8_43e8_8a90_9c0cc539dc44.slice/crio-8cdf2dbf0e937441cd27066eaf4a1278e5bc6bca7f57a0536b788cd9dc16a641 WatchSource:0}: Error finding container 8cdf2dbf0e937441cd27066eaf4a1278e5bc6bca7f57a0536b788cd9dc16a641: Status 404 returned error can't find the container with id 8cdf2dbf0e937441cd27066eaf4a1278e5bc6bca7f57a0536b788cd9dc16a641
Oct 30 19:23:23.588405 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:23.588385    6014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:23:24.409983 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:24.409939    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-994975fdf-wp825" event={"ID":"e7961869-97b8-43e8-8a90-9c0cc539dc44","Type":"ContainerStarted","Data":"8cdf2dbf0e937441cd27066eaf4a1278e5bc6bca7f57a0536b788cd9dc16a641"}
Oct 30 19:23:27.449568 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:27.449525    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-994975fdf-wp825" event={"ID":"e7961869-97b8-43e8-8a90-9c0cc539dc44","Type":"ContainerStarted","Data":"dcb68702a875bf90e05c24a30eb336a24376afe9e3ad4a76edd2cdbe058453ab"}
Oct 30 19:23:27.449995 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:27.449730    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:27.563813 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:27.563756    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-994975fdf-wp825" podStartSLOduration=2.540369527 podStartE2EDuration="5.563741537s" totalImagesPullingTime="3.02337201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:27.562918142 +0000 UTC m=+2646.206085756" watchObservedRunningTime="2026-10-30 19:23:27.563741537 +0000 UTC m=+2646.206909151"
Oct 30 19:23:34.047868 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:34.047832    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"]
Oct 30 19:23:34.097506 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:34.097471    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c1fc0e-9pp9h"]
Oct 30 19:23:35.475324 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:35.475292    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ea8cb62-852f-4bd1-b948-218ec0b214e6" path="/var/lib/kubelet/pods/9ea8cb62-852f-4bd1-b948-218ec0b214e6/volumes"
Oct 30 19:23:38.459621 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:38.459589    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:53.258703 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:23:53.258663    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:23:58.989747 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:58.989709    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-994975fdf-wp825"]
Oct 30 19:23:58.990420 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:58.989995    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-994975fdf-wp825" podUID="e7961869-97b8-43e8-8a90-9c0cc539dc44" containerName="manager" containerID="cri-o://dcb68702a875bf90e05c24a30eb336a24376afe9e3ad4a76edd2cdbe058453ab" gracePeriod=30
Oct 30 19:23:59.252425 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.252355    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:59.301719 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.301680    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-webhook-cert\") pod \"e7961869-97b8-43e8-8a90-9c0cc539dc44\" (UID: \"e7961869-97b8-43e8-8a90-9c0cc539dc44\") "
Oct 30 19:23:59.301878 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.301808    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7961869-97b8-43e8-8a90-9c0cc539dc44-kube-api-access-rmb8b\" (UniqueName: \"kubernetes.io/projected/e7961869-97b8-43e8-8a90-9c0cc539dc44-kube-api-access-rmb8b\") pod \"e7961869-97b8-43e8-8a90-9c0cc539dc44\" (UID: \"e7961869-97b8-43e8-8a90-9c0cc539dc44\") "
Oct 30 19:23:59.301878 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.301853    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-metrics-certs\") pod \"e7961869-97b8-43e8-8a90-9c0cc539dc44\" (UID: \"e7961869-97b8-43e8-8a90-9c0cc539dc44\") "
Oct 30 19:23:59.304162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.304108    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7961869-97b8-43e8-8a90-9c0cc539dc44-kube-api-access-rmb8b" pod "e7961869-97b8-43e8-8a90-9c0cc539dc44" (UID: "e7961869-97b8-43e8-8a90-9c0cc539dc44"). InnerVolumeSpecName "kube-api-access-rmb8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:59.304162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.304112    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-webhook-cert" pod "e7961869-97b8-43e8-8a90-9c0cc539dc44" (UID: "e7961869-97b8-43e8-8a90-9c0cc539dc44"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:59.304162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.304125    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-metrics-certs" pod "e7961869-97b8-43e8-8a90-9c0cc539dc44" (UID: "e7961869-97b8-43e8-8a90-9c0cc539dc44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:59.403027 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.402996    6014 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-webhook-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:59.403027 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.403025    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmb8b\" (UniqueName: \"kubernetes.io/projected/e7961869-97b8-43e8-8a90-9c0cc539dc44-kube-api-access-rmb8b\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:59.403027 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.403035    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7961869-97b8-43e8-8a90-9c0cc539dc44-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:59.635059 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.635023    6014 generic.go:356] "Generic (PLEG): container finished" podID="e7961869-97b8-43e8-8a90-9c0cc539dc44" containerID="dcb68702a875bf90e05c24a30eb336a24376afe9e3ad4a76edd2cdbe058453ab" exitCode=0
Oct 30 19:23:59.635281 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.635067    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-994975fdf-wp825"
Oct 30 19:23:59.635281 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.635105    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-994975fdf-wp825" event={"ID":"e7961869-97b8-43e8-8a90-9c0cc539dc44","Type":"ContainerDied","Data":"dcb68702a875bf90e05c24a30eb336a24376afe9e3ad4a76edd2cdbe058453ab"}
Oct 30 19:23:59.635281 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.635195    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-994975fdf-wp825" event={"ID":"e7961869-97b8-43e8-8a90-9c0cc539dc44","Type":"ContainerDied","Data":"8cdf2dbf0e937441cd27066eaf4a1278e5bc6bca7f57a0536b788cd9dc16a641"}
Oct 30 19:23:59.635281 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.635222    6014 scope.go:122] "RemoveContainer" containerID="dcb68702a875bf90e05c24a30eb336a24376afe9e3ad4a76edd2cdbe058453ab"
Oct 30 19:23:59.644456 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.644435    6014 scope.go:122] "RemoveContainer" containerID="dcb68702a875bf90e05c24a30eb336a24376afe9e3ad4a76edd2cdbe058453ab"
Oct 30 19:23:59.644752 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:23:59.644731    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb68702a875bf90e05c24a30eb336a24376afe9e3ad4a76edd2cdbe058453ab\": container with ID starting with dcb68702a875bf90e05c24a30eb336a24376afe9e3ad4a76edd2cdbe058453ab not found: ID does not exist" containerID="dcb68702a875bf90e05c24a30eb336a24376afe9e3ad4a76edd2cdbe058453ab"
Oct 30 19:23:59.644869 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.644760    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb68702a875bf90e05c24a30eb336a24376afe9e3ad4a76edd2cdbe058453ab"} err="failed to get container status \"dcb68702a875bf90e05c24a30eb336a24376afe9e3ad4a76edd2cdbe058453ab\": rpc error: code = NotFound desc = could not find container \"dcb68702a875bf90e05c24a30eb336a24376afe9e3ad4a76edd2cdbe058453ab\": container with ID starting with dcb68702a875bf90e05c24a30eb336a24376afe9e3ad4a76edd2cdbe058453ab not found: ID does not exist"
Oct 30 19:23:59.798473 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.798436    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-994975fdf-wp825"]
Oct 30 19:23:59.847949 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:23:59.847917    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-994975fdf-wp825"]
Oct 30 19:24:01.213996 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.213960    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-994975fdf-hb6nt"]
Oct 30 19:24:01.214489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.214467    6014 cpu_manager.go:414] "containerMap: removing container" podUID="e7961869-97b8-43e8-8a90-9c0cc539dc44" containerName="manager"
Oct 30 19:24:01.214489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.214479    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7961869-97b8-43e8-8a90-9c0cc539dc44" containerName="manager"
Oct 30 19:24:01.214577 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.214546    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7961869-97b8-43e8-8a90-9c0cc539dc44" containerName="manager"
Oct 30 19:24:01.218164 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.218130    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-994975fdf-hb6nt"
Oct 30 19:24:01.320326 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.320284    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 19:24:01.320326 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.320305    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 19:24:01.320326 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.320294    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wwn5x\""
Oct 30 19:24:01.320675 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.320352    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 19:24:01.324841 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.324787    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6gdg\" (UniqueName: \"kubernetes.io/projected/4405e9f5-ccac-4971-84f7-24612c1de6c2-kube-api-access-h6gdg\") pod \"maas-controller-994975fdf-hb6nt\" (UID: \"4405e9f5-ccac-4971-84f7-24612c1de6c2\") " pod="opendatahub/maas-controller-994975fdf-hb6nt"
Oct 30 19:24:01.325032 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.324870    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4405e9f5-ccac-4971-84f7-24612c1de6c2-webhook-cert\") pod \"maas-controller-994975fdf-hb6nt\" (UID: \"4405e9f5-ccac-4971-84f7-24612c1de6c2\") " pod="opendatahub/maas-controller-994975fdf-hb6nt"
Oct 30 19:24:01.325032 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.324943    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4405e9f5-ccac-4971-84f7-24612c1de6c2-metrics-certs\") pod \"maas-controller-994975fdf-hb6nt\" (UID: \"4405e9f5-ccac-4971-84f7-24612c1de6c2\") " pod="opendatahub/maas-controller-994975fdf-hb6nt"
Oct 30 19:24:01.325274 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.325253    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-994975fdf-hb6nt"]
Oct 30 19:24:01.425868 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.425829    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6gdg\" (UniqueName: \"kubernetes.io/projected/4405e9f5-ccac-4971-84f7-24612c1de6c2-kube-api-access-h6gdg\") pod \"maas-controller-994975fdf-hb6nt\" (UID: \"4405e9f5-ccac-4971-84f7-24612c1de6c2\") " pod="opendatahub/maas-controller-994975fdf-hb6nt"
Oct 30 19:24:01.426106 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.425900    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4405e9f5-ccac-4971-84f7-24612c1de6c2-webhook-cert\") pod \"maas-controller-994975fdf-hb6nt\" (UID: \"4405e9f5-ccac-4971-84f7-24612c1de6c2\") " pod="opendatahub/maas-controller-994975fdf-hb6nt"
Oct 30 19:24:01.426106 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.425959    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4405e9f5-ccac-4971-84f7-24612c1de6c2-metrics-certs\") pod \"maas-controller-994975fdf-hb6nt\" (UID: \"4405e9f5-ccac-4971-84f7-24612c1de6c2\") " pod="opendatahub/maas-controller-994975fdf-hb6nt"
Oct 30 19:24:01.428647 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.428614    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4405e9f5-ccac-4971-84f7-24612c1de6c2-metrics-certs\") pod \"maas-controller-994975fdf-hb6nt\" (UID: \"4405e9f5-ccac-4971-84f7-24612c1de6c2\") " pod="opendatahub/maas-controller-994975fdf-hb6nt"
Oct 30 19:24:01.428753 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.428623    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4405e9f5-ccac-4971-84f7-24612c1de6c2-webhook-cert\") pod \"maas-controller-994975fdf-hb6nt\" (UID: \"4405e9f5-ccac-4971-84f7-24612c1de6c2\") " pod="opendatahub/maas-controller-994975fdf-hb6nt"
Oct 30 19:24:01.474956 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.474869    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7961869-97b8-43e8-8a90-9c0cc539dc44" path="/var/lib/kubelet/pods/e7961869-97b8-43e8-8a90-9c0cc539dc44/volumes"
Oct 30 19:24:01.484743 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.484711    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6gdg\" (UniqueName: \"kubernetes.io/projected/4405e9f5-ccac-4971-84f7-24612c1de6c2-kube-api-access-h6gdg\") pod \"maas-controller-994975fdf-hb6nt\" (UID: \"4405e9f5-ccac-4971-84f7-24612c1de6c2\") " pod="opendatahub/maas-controller-994975fdf-hb6nt"
Oct 30 19:24:01.532233 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.532194    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-994975fdf-hb6nt"
Oct 30 19:24:01.920962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:01.920928    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-994975fdf-hb6nt"]
Oct 30 19:24:01.925034 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:24:01.925005    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4405e9f5_ccac_4971_84f7_24612c1de6c2.slice/crio-61dfbec8735eec991fff735026d9fbaceddd1e8585e8dfe683f02635f0858ee6 WatchSource:0}: Error finding container 61dfbec8735eec991fff735026d9fbaceddd1e8585e8dfe683f02635f0858ee6: Status 404 returned error can't find the container with id 61dfbec8735eec991fff735026d9fbaceddd1e8585e8dfe683f02635f0858ee6
Oct 30 19:24:02.654572 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:02.654536    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-994975fdf-hb6nt" event={"ID":"4405e9f5-ccac-4971-84f7-24612c1de6c2","Type":"ContainerStarted","Data":"61dfbec8735eec991fff735026d9fbaceddd1e8585e8dfe683f02635f0858ee6"}
Oct 30 19:24:03.660512 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:03.660471    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-994975fdf-hb6nt" event={"ID":"4405e9f5-ccac-4971-84f7-24612c1de6c2","Type":"ContainerStarted","Data":"5878f23aa1644d491892d059051e3f6a69591644ee0a9963790b7e991088fa51"}
Oct 30 19:24:03.660920 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:03.660602    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-994975fdf-hb6nt"
Oct 30 19:24:03.775377 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:03.775321    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-994975fdf-hb6nt" podStartSLOduration=1.872578745 podStartE2EDuration="2.77530523s" totalImagesPullingTime="902.726485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:03.7741862 +0000 UTC m=+2682.417353814" watchObservedRunningTime="2026-10-30 19:24:03.77530523 +0000 UTC m=+2682.418472846"
Oct 30 19:24:14.669236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:14.669203    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-994975fdf-hb6nt"
Oct 30 19:24:23.259481 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:24:23.259440    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:24:24.974159 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:24.974115    6014 scope.go:122] "RemoveContainer" containerID="4b2e25c6a5b5c69fa72010a00a153511989c7ba12154d43e5af1b1995cae1887"
Oct 30 19:24:24.983126 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:24.983104    6014 scope.go:122] "RemoveContainer" containerID="0a5af15919b60336e29b194314907258f3f86f751dd3fbf05361084c81e65a58"
Oct 30 19:24:24.991292 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:24.991271    6014 scope.go:122] "RemoveContainer" containerID="5d3a3b2089aac09593448a67d3fc09c13fa146beca35d0d12b2ab594ebfc6822"
Oct 30 19:24:37.615444 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:37.615355    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:24:53.257838 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:24:53.257799    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:24:59.218125 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:24:59.218089    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:25:06.194167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:25:06.193274    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9r6z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:25:06.194167 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:25:06.193386    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2" podUID="b8882aef-7227-4781-8645-ed9c06928737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:08.413216 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:25:08.413178    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:25:23.257793 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:25:23.257752    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:25:25.022920 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:25:25.022883    6014 scope.go:122] "RemoveContainer" containerID="194ac092019fb7cb8f6596f238599cd0aeb167e825b0cf5bfa65dbaea0db2af4"
Oct 30 19:25:29.057436 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:25:29.057399    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:25:36.943872 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:25:36.943829    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:25:43.153778 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:25:43.153737    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:25:53.258266 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:25:53.258228    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:26:14.926778 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:26:14.926696    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:26:23.258424 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:26:23.258384    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:26:25.622717 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:26:25.622687    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:26:35.744056 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:26:35.744015    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:26:53.258104 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:26:53.258066    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:27:00.449071 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:27:00.449029    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:27:08.168095 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:27:08.168048    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:27:23.258243 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:27:23.258204    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:27:28.730247 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:27:28.730212    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:27:40.431980 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:27:40.431893    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:27:53.257212 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:27:53.257169    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:27:55.052809 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:27:55.052772    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"]
Oct 30 19:27:55.102162 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:27:55.102101    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13d524a-lrcdm"]
Oct 30 19:27:55.475966 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:27:55.475930    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d89e2e9-48ab-44c2-9fd4-6c616edb400e" path="/var/lib/kubelet/pods/9d89e2e9-48ab-44c2-9fd4-6c616edb400e/volumes"
Oct 30 19:28:00.748488 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:28:00.748446    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:28:03.165336 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:28:03.165276    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5s95d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:28:03.166043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:28:03.165968    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d" podUID="c357435d-3e86-4482-8419-a4050103c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:19.684491 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:28:19.684455    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:28:23.258225 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:28:23.258183    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:28:53.259407 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:28:53.259329    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:29:14.740823 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.740745    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"]
Oct 30 19:29:14.744553 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.744531    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.784988 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.784946    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6nrk\" (UniqueName: \"kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-kube-api-access-l6nrk\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.785311 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.785286    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istiod-ca-cert\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.785519 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.785503    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-certs\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.785630 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.785617    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-envoy\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.785771 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.785757    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-socket\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.785879 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.785866    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-token\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.785981 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.785966    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-data\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.786108 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.786073    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-credential-socket\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.786287 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.786128    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-podinfo\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.796489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.796452    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-da238b-openshift-default-dockercfg-n9d9g\""
Oct 30 19:29:14.848276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.848239    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"]
Oct 30 19:29:14.887125 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.887089    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-podinfo\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.887369 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.887219    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6nrk\" (UniqueName: \"kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-kube-api-access-l6nrk\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.887369 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.887253    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istiod-ca-cert\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.887369 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.887277    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-certs\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.887369 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.887292    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-envoy\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.887369 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.887334    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-socket\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.887369 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.887362    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-token\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.887648 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.887405    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-data\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.887648 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.887434    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-credential-socket\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.887846 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.887791    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-socket\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.888004 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.887841    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-certs\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.888004 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.887852    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-credential-socket\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.888004 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.887928    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-data\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.888736 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.888713    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istiod-ca-cert\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.889990 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.889965    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-envoy\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.890257 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.890233    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-podinfo\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.944372 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.944343    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-token\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:14.944686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:14.944665    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6nrk\" (UniqueName: \"kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-kube-api-access-l6nrk\") pod \"e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") " pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:15.058194 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:15.058081    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:15.491089 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:15.491014    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"]
Oct 30 19:29:15.493632 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:29:15.493598    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a7e776_ade5_45f9_aa6e_1ee581ef37e7.slice/crio-c93ee56281ef79dae83fd8bcf275649f5cf5860a417c91eba3416ec0858abe2e WatchSource:0}: Error finding container c93ee56281ef79dae83fd8bcf275649f5cf5860a417c91eba3416ec0858abe2e: Status 404 returned error can't find the container with id c93ee56281ef79dae83fd8bcf275649f5cf5860a417c91eba3416ec0858abe2e
Oct 30 19:29:15.495850 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:15.495811    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:29:15.495973 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:15.495897    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:29:15.495973 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:15.495943    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:29:16.402793 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:16.402754    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" event={"ID":"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7","Type":"ContainerStarted","Data":"1d5897e14dbc0e3927fc2fb7a8243e720b71b3130b493a2d1a18cd7998a7b569"}
Oct 30 19:29:16.403242 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:16.402813    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" event={"ID":"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7","Type":"ContainerStarted","Data":"c93ee56281ef79dae83fd8bcf275649f5cf5860a417c91eba3416ec0858abe2e"}
Oct 30 19:29:16.620135 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:16.620075    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" podStartSLOduration=2.62006289 podStartE2EDuration="2.62006289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:16.569785554 +0000 UTC m=+2995.212953168" watchObservedRunningTime="2026-10-30 19:29:16.62006289 +0000 UTC m=+2995.263230503"
Oct 30 19:29:17.059219 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:17.059176    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:17.064245 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:17.064218    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:17.408542 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:17.408509    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:17.409652 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:17.409635    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:23.265005 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:29:23.264966    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:29:25.068963 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.068925    6014 scope.go:122] "RemoveContainer" containerID="36c4f72d57c4a7a90e9e7eff7ac41a1768d205f5298d9e72adc88bc60976f561"
Oct 30 19:29:25.084931 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.084909    6014 scope.go:122] "RemoveContainer" containerID="9ae9230c2d401aa04fae75b68b6fdbd25a5de4e2fdac862580c8f64c036f54a2"
Oct 30 19:29:25.440003 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.439901    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"]
Oct 30 19:29:25.447636 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.447601    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:25.499970 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.499922    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:29:25.505549 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.505506    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhdxv\" (UniqueName: \"kubernetes.io/projected/8d544589-1156-4e83-91d4-306137cb5bb1-kube-api-access-mhdxv\") pod \"maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:25.505762 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.505731    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d544589-1156-4e83-91d4-306137cb5bb1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:25.505850 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.505783    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:25.505957 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.505930    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:25.549498 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.549453    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-da238b-serving-cert\""
Oct 30 19:29:25.549686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.549523    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:29:25.549686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.549598    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9wflp\""
Oct 30 19:29:25.549782 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.549699    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:29:25.552047 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.552019    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"]
Oct 30 19:29:25.606998 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.606967    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhdxv\" (UniqueName: \"kubernetes.io/projected/8d544589-1156-4e83-91d4-306137cb5bb1-kube-api-access-mhdxv\") pod \"maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:25.607195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.607082    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d544589-1156-4e83-91d4-306137cb5bb1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:25.607195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.607113    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:25.607195 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.607133    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:25.607338 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:29:25.607279    6014 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-serving-cert: secret "maas-api-e2e-worker-single-main-da238b-serving-cert" not found
Oct 30 19:29:25.607377 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:29:25.607345    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-maas-api-tls podName:8d544589-1156-4e83-91d4-306137cb5bb1 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:26.107327232 +0000 UTC m=+3004.750494827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-maas-api-tls") pod "maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll" (UID: "8d544589-1156-4e83-91d4-306137cb5bb1") : secret "maas-api-e2e-worker-single-main-da238b-serving-cert" not found
Oct 30 19:29:25.608018 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.607991    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d544589-1156-4e83-91d4-306137cb5bb1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:25.609854 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.609833    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:25.664381 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:25.664350    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhdxv\" (UniqueName: \"kubernetes.io/projected/8d544589-1156-4e83-91d4-306137cb5bb1-kube-api-access-mhdxv\") pod \"maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:26.112797 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:26.112755    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:26.115555 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:26.115530    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:26.361461 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:26.361423    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:26.734277 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:26.734017    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"]
Oct 30 19:29:26.737311 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:29:26.737282    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d544589_1156_4e83_91d4_306137cb5bb1.slice/crio-ce40f7683a3670f1cbd0878dde4afe1907b700673cc8d396ffefdee859bd12d7 WatchSource:0}: Error finding container ce40f7683a3670f1cbd0878dde4afe1907b700673cc8d396ffefdee859bd12d7: Status 404 returned error can't find the container with id ce40f7683a3670f1cbd0878dde4afe1907b700673cc8d396ffefdee859bd12d7
Oct 30 19:29:26.738780 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:26.738759    6014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:29:27.463224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:27.463184    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll" event={"ID":"8d544589-1156-4e83-91d4-306137cb5bb1","Type":"ContainerStarted","Data":"ce40f7683a3670f1cbd0878dde4afe1907b700673cc8d396ffefdee859bd12d7"}
Oct 30 19:29:28.469867 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:28.469835    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll" event={"ID":"8d544589-1156-4e83-91d4-306137cb5bb1","Type":"ContainerStarted","Data":"7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c"}
Oct 30 19:29:28.470504 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:28.469966    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:28.585752 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:28.585692    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll" podStartSLOduration=2.736588103 podStartE2EDuration="3.585675848s" totalImagesPullingTime="849.087745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:28.584096183 +0000 UTC m=+3007.227263797" watchObservedRunningTime="2026-10-30 19:29:28.585675848 +0000 UTC m=+3007.228843461"
Oct 30 19:29:33.233621 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:33.233588    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:29:34.481877 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:34.481849    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:29:42.546102 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:42.546003    6014 generic.go:356] "Generic (PLEG): container finished" podID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerID="1d5897e14dbc0e3927fc2fb7a8243e720b71b3130b493a2d1a18cd7998a7b569" exitCode=0
Oct 30 19:29:42.546102 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:42.546070    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" event={"ID":"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7","Type":"ContainerDied","Data":"1d5897e14dbc0e3927fc2fb7a8243e720b71b3130b493a2d1a18cd7998a7b569"}
Oct 30 19:29:42.546611 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:42.546468    6014 scope.go:122] "RemoveContainer" containerID="1d5897e14dbc0e3927fc2fb7a8243e720b71b3130b493a2d1a18cd7998a7b569"
Oct 30 19:29:42.547706 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:42.547676    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:29:42.547769 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:42.547752    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:29:42.547812 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:42.547787    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:29:43.577453 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:43.577420    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" event={"ID":"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7","Type":"ContainerStarted","Data":"1e01e4da223eb9372c98488d24bfa4c8133aa204e06226719e79bec50c71ccc2"}
Oct 30 19:29:44.062634 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:44.062534    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:44.063064 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:44.063037    6014 patch_prober.go:29] interesting pod/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 19:29:44.063277 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:44.063250    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 19:29:45.059616 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:45.059572    6014 patch_prober.go:29] interesting pod/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 19:29:45.060271 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:45.060239    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 19:29:46.065965 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:46.065927    6014 patch_prober.go:29] interesting pod/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 19:29:46.066613 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:46.066580    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 19:29:47.063627 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:47.063585    6014 patch_prober.go:29] interesting pod/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 19:29:47.063962 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:47.063931    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 19:29:47.408955 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:47.408867    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:48.072656 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:48.072622    6014 patch_prober.go:29] interesting pod/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 19:29:48.072953 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:48.072922    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 19:29:49.062753 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:49.062708    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:49.063690 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:49.063673    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:29:52.534270 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:29:52.534238    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:29:53.258108 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:29:53.258072    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:30:16.914781 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:16.914742    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:30:23.257693 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:30:23.257657    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:30:37.232912 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:37.232834    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:30:51.227912 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:51.227872    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"]
Oct 30 19:30:51.228537 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:51.228096    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll" podUID="8d544589-1156-4e83-91d4-306137cb5bb1" containerName="maas-api" containerID="cri-o://7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c" gracePeriod=30
Oct 30 19:30:52.372320 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:30:52.372284    6014 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d544589_1156_4e83_91d4_306137cb5bb1.slice/crio-conmon-7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d544589_1156_4e83_91d4_306137cb5bb1.slice/crio-7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:30:52.504447 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.504420    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:30:52.604472 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.604370    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-maas-api-tls\") pod \"8d544589-1156-4e83-91d4-306137cb5bb1\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") "
Oct 30 19:30:52.604472 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.604439    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d544589-1156-4e83-91d4-306137cb5bb1-kube-api-access-mhdxv\" (UniqueName: \"kubernetes.io/projected/8d544589-1156-4e83-91d4-306137cb5bb1-kube-api-access-mhdxv\") pod \"8d544589-1156-4e83-91d4-306137cb5bb1\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") "
Oct 30 19:30:52.604718 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.604569    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-metrics-certs\") pod \"8d544589-1156-4e83-91d4-306137cb5bb1\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") "
Oct 30 19:30:52.604718 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.604653    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d544589-1156-4e83-91d4-306137cb5bb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d544589-1156-4e83-91d4-306137cb5bb1-openshift-service-ca\") pod \"8d544589-1156-4e83-91d4-306137cb5bb1\" (UID: \"8d544589-1156-4e83-91d4-306137cb5bb1\") "
Oct 30 19:30:52.605112 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.605082    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d544589-1156-4e83-91d4-306137cb5bb1-openshift-service-ca" pod "8d544589-1156-4e83-91d4-306137cb5bb1" (UID: "8d544589-1156-4e83-91d4-306137cb5bb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:52.606862 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.606834    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-metrics-certs" pod "8d544589-1156-4e83-91d4-306137cb5bb1" (UID: "8d544589-1156-4e83-91d4-306137cb5bb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:52.606975 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.606854    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-maas-api-tls" pod "8d544589-1156-4e83-91d4-306137cb5bb1" (UID: "8d544589-1156-4e83-91d4-306137cb5bb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:52.607029 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.607001    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d544589-1156-4e83-91d4-306137cb5bb1-kube-api-access-mhdxv" pod "8d544589-1156-4e83-91d4-306137cb5bb1" (UID: "8d544589-1156-4e83-91d4-306137cb5bb1"). InnerVolumeSpecName "kube-api-access-mhdxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:52.706125 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.706086    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d544589-1156-4e83-91d4-306137cb5bb1-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:52.706125 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.706118    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:52.706125 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.706131    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhdxv\" (UniqueName: \"kubernetes.io/projected/8d544589-1156-4e83-91d4-306137cb5bb1-kube-api-access-mhdxv\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:52.706406 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.706167    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d544589-1156-4e83-91d4-306137cb5bb1-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:52.937380 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.937278    6014 generic.go:356] "Generic (PLEG): container finished" podID="8d544589-1156-4e83-91d4-306137cb5bb1" containerID="7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c" exitCode=0
Oct 30 19:30:52.937380 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.937330    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"
Oct 30 19:30:52.937622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.937372    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll" event={"ID":"8d544589-1156-4e83-91d4-306137cb5bb1","Type":"ContainerDied","Data":"7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c"}
Oct 30 19:30:52.937622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.937453    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll" event={"ID":"8d544589-1156-4e83-91d4-306137cb5bb1","Type":"ContainerDied","Data":"ce40f7683a3670f1cbd0878dde4afe1907b700673cc8d396ffefdee859bd12d7"}
Oct 30 19:30:52.937622 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.937476    6014 scope.go:122] "RemoveContainer" containerID="7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c"
Oct 30 19:30:52.947282 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.947266    6014 scope.go:122] "RemoveContainer" containerID="7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c"
Oct 30 19:30:52.947569 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:30:52.947550    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c\": container with ID starting with 7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c not found: ID does not exist" containerID="7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c"
Oct 30 19:30:52.947628 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:52.947579    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c"} err="failed to get container status \"7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c\": rpc error: code = NotFound desc = could not find container \"7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c\": container with ID starting with 7e30acfb62ade233713788c3fd5e501038c2c7a849eefcf7e9f1ded1be695f5c not found: ID does not exist"
Oct 30 19:30:53.103680 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:53.103640    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"]
Oct 30 19:30:53.153709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:53.153673    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-da238b-67dc8544ff-vqvll"]
Oct 30 19:30:53.258369 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:30:53.258279    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:30:53.475479 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:53.475444    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d544589-1156-4e83-91d4-306137cb5bb1" path="/var/lib/kubelet/pods/8d544589-1156-4e83-91d4-306137cb5bb1/volumes"
Oct 30 19:30:55.766251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:30:55.766214    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:31:00.455267 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:00.455228    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"]
Oct 30 19:31:00.455758 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:00.455545    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerName="istio-proxy" containerID="cri-o://1e01e4da223eb9372c98488d24bfa4c8133aa204e06226719e79bec50c71ccc2" gracePeriod=30
Oct 30 19:31:04.066087 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:04.066053    6014 patch_prober.go:29] interesting pod/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:31:04.066492 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:04.066114    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:31:05.711738 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.711712    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:31:05.841608 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.841577    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-kube-api-access-l6nrk\" (UniqueName: \"kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-kube-api-access-l6nrk\") pod \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") "
Oct 30 19:31:05.841769 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.841647    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-socket\") pod \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") "
Oct 30 19:31:05.841769 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.841698    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-token\" (UniqueName: \"kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-token\") pod \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") "
Oct 30 19:31:05.841868 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.841777    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-envoy\") pod \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") "
Oct 30 19:31:05.841934 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.841910    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istiod-ca-cert\") pod \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") "
Oct 30 19:31:05.842020 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.841985    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-certs\") pod \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") "
Oct 30 19:31:05.842131 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.842063    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-data\") pod \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") "
Oct 30 19:31:05.842131 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.842121    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-credential-socket\") pod \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") "
Oct 30 19:31:05.842291 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.842192    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-podinfo\") pod \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\" (UID: \"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7\") "
Oct 30 19:31:05.842353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.841988    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-socket" pod "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" (UID: "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:05.842353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.842241    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-certs" pod "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" (UID: "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:05.842353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.842308    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istiod-ca-cert" pod "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" (UID: "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:05.842353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.842308    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-data" pod "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" (UID: "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:05.842568 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.842497    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-credential-socket" pod "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" (UID: "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:05.842827 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.842801    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.842886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.842830    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.842886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.842846    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.842886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.842859    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.842886 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.842873    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.844247 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.844199    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-token" pod "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" (UID: "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:05.844500 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.844480    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-podinfo" pod "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" (UID: "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:31:05.844600 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.844580    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-envoy" pod "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" (UID: "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:05.844642 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.844599    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-kube-api-access-l6nrk" pod "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" (UID: "f5a7e776-ade5-45f9-aa6e-1ee581ef37e7"). InnerVolumeSpecName "kube-api-access-l6nrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:05.943799 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.943763    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.943799 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.943793    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.943799 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.943803    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.944055 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:05.943815    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6nrk\" (UniqueName: \"kubernetes.io/projected/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7-kube-api-access-l6nrk\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:06.022039 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:06.022008    6014 generic.go:356] "Generic (PLEG): container finished" podID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerID="1e01e4da223eb9372c98488d24bfa4c8133aa204e06226719e79bec50c71ccc2" exitCode=0
Oct 30 19:31:06.022272 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:06.022063    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"
Oct 30 19:31:06.022272 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:06.022114    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" event={"ID":"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7","Type":"ContainerDied","Data":"1e01e4da223eb9372c98488d24bfa4c8133aa204e06226719e79bec50c71ccc2"}
Oct 30 19:31:06.022272 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:06.022192    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4" event={"ID":"f5a7e776-ade5-45f9-aa6e-1ee581ef37e7","Type":"ContainerDied","Data":"c93ee56281ef79dae83fd8bcf275649f5cf5860a417c91eba3416ec0858abe2e"}
Oct 30 19:31:06.022272 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:06.022216    6014 scope.go:122] "RemoveContainer" containerID="1e01e4da223eb9372c98488d24bfa4c8133aa204e06226719e79bec50c71ccc2"
Oct 30 19:31:06.032161 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:06.032125    6014 scope.go:122] "RemoveContainer" containerID="1d5897e14dbc0e3927fc2fb7a8243e720b71b3130b493a2d1a18cd7998a7b569"
Oct 30 19:31:06.040500 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:06.040477    6014 scope.go:122] "RemoveContainer" containerID="1e01e4da223eb9372c98488d24bfa4c8133aa204e06226719e79bec50c71ccc2"
Oct 30 19:31:06.040749 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:31:06.040729    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e01e4da223eb9372c98488d24bfa4c8133aa204e06226719e79bec50c71ccc2\": container with ID starting with 1e01e4da223eb9372c98488d24bfa4c8133aa204e06226719e79bec50c71ccc2 not found: ID does not exist" containerID="1e01e4da223eb9372c98488d24bfa4c8133aa204e06226719e79bec50c71ccc2"
Oct 30 19:31:06.040835 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:06.040763    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e01e4da223eb9372c98488d24bfa4c8133aa204e06226719e79bec50c71ccc2"} err="failed to get container status \"1e01e4da223eb9372c98488d24bfa4c8133aa204e06226719e79bec50c71ccc2\": rpc error: code = NotFound desc = could not find container \"1e01e4da223eb9372c98488d24bfa4c8133aa204e06226719e79bec50c71ccc2\": container with ID starting with 1e01e4da223eb9372c98488d24bfa4c8133aa204e06226719e79bec50c71ccc2 not found: ID does not exist"
Oct 30 19:31:06.040835 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:06.040791    6014 scope.go:122] "RemoveContainer" containerID="1d5897e14dbc0e3927fc2fb7a8243e720b71b3130b493a2d1a18cd7998a7b569"
Oct 30 19:31:06.041043 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:31:06.041026    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d5897e14dbc0e3927fc2fb7a8243e720b71b3130b493a2d1a18cd7998a7b569\": container with ID starting with 1d5897e14dbc0e3927fc2fb7a8243e720b71b3130b493a2d1a18cd7998a7b569 not found: ID does not exist" containerID="1d5897e14dbc0e3927fc2fb7a8243e720b71b3130b493a2d1a18cd7998a7b569"
Oct 30 19:31:06.041099 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:06.041054    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d5897e14dbc0e3927fc2fb7a8243e720b71b3130b493a2d1a18cd7998a7b569"} err="failed to get container status \"1d5897e14dbc0e3927fc2fb7a8243e720b71b3130b493a2d1a18cd7998a7b569\": rpc error: code = NotFound desc = could not find container \"1d5897e14dbc0e3927fc2fb7a8243e720b71b3130b493a2d1a18cd7998a7b569\": container with ID starting with 1d5897e14dbc0e3927fc2fb7a8243e720b71b3130b493a2d1a18cd7998a7b569 not found: ID does not exist"
Oct 30 19:31:06.188829 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:06.188744    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"]
Oct 30 19:31:06.195328 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:06.195299    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-da238b-openshift-default-744b7f9fdf7jkc4"]
Oct 30 19:31:07.474866 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:31:07.474833    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" path="/var/lib/kubelet/pods/f5a7e776-ade5-45f9-aa6e-1ee581ef37e7/volumes"
Oct 30 19:31:23.258017 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:31:23.257980    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:31:53.258111 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:31:53.258069    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:32:23.258074 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:32:23.257974    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:32:47.151291 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:32:47.151252    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:32:53.258309 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:32:53.258263    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:33:04.509293 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:04.509258    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:33:11.879371 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:11.879322    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dmh57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:33:11.879829 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:11.879408    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" podUID="aacb99a2-5355-4e64-86bb-ad338415035f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:23.257827 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:33:23.257788    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:33:26.226863 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.226825    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"]
Oct 30 19:33:26.227356 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.227342    6014 cpu_manager.go:414] "containerMap: removing container" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerName="istio-proxy"
Oct 30 19:33:26.227410 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.227357    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerName="istio-proxy"
Oct 30 19:33:26.227410 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.227377    6014 cpu_manager.go:414] "containerMap: removing container" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerName="istio-proxy"
Oct 30 19:33:26.227410 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.227382    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerName="istio-proxy"
Oct 30 19:33:26.227410 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.227401    6014 cpu_manager.go:414] "containerMap: removing container" podUID="8d544589-1156-4e83-91d4-306137cb5bb1" containerName="maas-api"
Oct 30 19:33:26.227410 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.227408    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d544589-1156-4e83-91d4-306137cb5bb1" containerName="maas-api"
Oct 30 19:33:26.227581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.227471    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerName="istio-proxy"
Oct 30 19:33:26.227581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.227481    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a7e776-ade5-45f9-aa6e-1ee581ef37e7" containerName="istio-proxy"
Oct 30 19:33:26.227581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.227488    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d544589-1156-4e83-91d4-306137cb5bb1" containerName="maas-api"
Oct 30 19:33:26.230589 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.230567    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.258861 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.258825    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps865\" (UniqueName: \"kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-kube-api-access-ps865\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.259075 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.258890    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-token\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.259075 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.259008    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-credential-socket\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.259075 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.259056    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-data\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.259240 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.259198    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-certs\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.259285 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.259245    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-socket\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.259358 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.259339    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3e59b5-20c8-48dc-870f-3382350d32fd-istiod-ca-cert\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.259412 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.259401    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-envoy\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.259466 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.259433    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-podinfo\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.282894 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.282857    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-682a87-openshift-default-dockercfg-jq2kt\""
Oct 30 19:33:26.336404 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.336365    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"]
Oct 30 19:33:26.360529 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.360490    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-envoy\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.360794 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.360531    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-podinfo\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.360794 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.360576    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps865\" (UniqueName: \"kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-kube-api-access-ps865\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.360794 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.360613    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-token\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.360794 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.360647    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-credential-socket\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.360794 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.360671    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-data\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.360794 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.360707    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-certs\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.360794 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.360730    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-socket\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.360794 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.360761    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3e59b5-20c8-48dc-870f-3382350d32fd-istiod-ca-cert\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.361371 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.361352    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3e59b5-20c8-48dc-870f-3382350d32fd-istiod-ca-cert\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.363918 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.362261    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-credential-socket\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.365616 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.364478    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-envoy\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.365616 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.364718    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-data\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.365616 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.364924    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-certs\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.365616 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.365122    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-socket\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.369707 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.369679    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-podinfo\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.422619 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.422584    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-token\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.422932 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.422906    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps865\" (UniqueName: \"kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-kube-api-access-ps865\") pod \"e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") " pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.543832 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.543750    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:26.970880 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.970852    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"]
Oct 30 19:33:26.972985 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:33:26.972954    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3e59b5_20c8_48dc_870f_3382350d32fd.slice/crio-5ba233df6bc2c0ce04d6b13d2f41e36664a940c2dde3a5bf82456c8d94124f42 WatchSource:0}: Error finding container 5ba233df6bc2c0ce04d6b13d2f41e36664a940c2dde3a5bf82456c8d94124f42: Status 404 returned error can't find the container with id 5ba233df6bc2c0ce04d6b13d2f41e36664a940c2dde3a5bf82456c8d94124f42
Oct 30 19:33:26.975283 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.975249    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:33:26.975379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.975313    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:33:26.975379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:26.975343    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:33:27.774322 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:27.774278    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj" event={"ID":"1b3e59b5-20c8-48dc-870f-3382350d32fd","Type":"ContainerStarted","Data":"5886834abaf0027a3dcf3e15e36fe71a9496fc0a807042dcd1213e00ba503ab7"}
Oct 30 19:33:27.774826 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:27.774334    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj" event={"ID":"1b3e59b5-20c8-48dc-870f-3382350d32fd","Type":"ContainerStarted","Data":"5ba233df6bc2c0ce04d6b13d2f41e36664a940c2dde3a5bf82456c8d94124f42"}
Oct 30 19:33:27.893267 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:27.893205    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj" podStartSLOduration=1.893189132 podStartE2EDuration="1.893189132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:27.89233252 +0000 UTC m=+3246.535500135" watchObservedRunningTime="2026-10-30 19:33:27.893189132 +0000 UTC m=+3246.536356745"
Oct 30 19:33:28.544664 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:28.544627    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:28.550201 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:28.550178    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:28.779821 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:28.779787    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:28.780734 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:28.780715    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:36.685199 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.685099    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"]
Oct 30 19:33:36.689216 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.689195    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:36.741671 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.741641    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-682a87-serving-cert\""
Oct 30 19:33:36.741671 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.741673    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:33:36.777674 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.777637    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhx55\" (UniqueName: \"kubernetes.io/projected/f049b708-83f0-4e71-8964-f9b49ebb78c4-kube-api-access-dhx55\") pod \"maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:36.777893 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.777700    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:36.777893 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.777848    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f049b708-83f0-4e71-8964-f9b49ebb78c4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:36.778041 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.777913    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:36.790877 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.790847    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:33:36.790877 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.790848    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:33:36.791116 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.790880    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9wflp\""
Oct 30 19:33:36.792865 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.792835    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"]
Oct 30 19:33:36.878560 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.878524    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f049b708-83f0-4e71-8964-f9b49ebb78c4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:36.878746 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.878615    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:36.878746 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.878674    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhx55\" (UniqueName: \"kubernetes.io/projected/f049b708-83f0-4e71-8964-f9b49ebb78c4-kube-api-access-dhx55\") pod \"maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:36.878746 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.878741    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:36.878875 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:33:36.878770    6014 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-serving-cert: secret "maas-api-e2e-worker-single-main-682a87-serving-cert" not found
Oct 30 19:33:36.878875 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:33:36.878843    6014 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-maas-api-tls podName:f049b708-83f0-4e71-8964-f9b49ebb78c4 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:37.378826145 +0000 UTC m=+3256.021993737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-maas-api-tls") pod "maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq" (UID: "f049b708-83f0-4e71-8964-f9b49ebb78c4") : secret "maas-api-e2e-worker-single-main-682a87-serving-cert" not found
Oct 30 19:33:36.879364 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.879340    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f049b708-83f0-4e71-8964-f9b49ebb78c4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:36.881393 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.881372    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:36.937198 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:36.937101    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhx55\" (UniqueName: \"kubernetes.io/projected/f049b708-83f0-4e71-8964-f9b49ebb78c4-kube-api-access-dhx55\") pod \"maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:37.384073 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:37.384031    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:37.386746 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:37.386716    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:37.603277 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:37.603243    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:37.975524 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:37.975443    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"]
Oct 30 19:33:37.978936 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:33:37.978907    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf049b708_83f0_4e71_8964_f9b49ebb78c4.slice/crio-0faa1f0db8453e537f306c9e12631ed4d83b385d14ea8911e7e4df7f06c416fa WatchSource:0}: Error finding container 0faa1f0db8453e537f306c9e12631ed4d83b385d14ea8911e7e4df7f06c416fa: Status 404 returned error can't find the container with id 0faa1f0db8453e537f306c9e12631ed4d83b385d14ea8911e7e4df7f06c416fa
Oct 30 19:33:38.843646 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:38.843603    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq" event={"ID":"f049b708-83f0-4e71-8964-f9b49ebb78c4","Type":"ContainerStarted","Data":"0faa1f0db8453e537f306c9e12631ed4d83b385d14ea8911e7e4df7f06c416fa"}
Oct 30 19:33:39.850101 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:39.850056    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq" event={"ID":"f049b708-83f0-4e71-8964-f9b49ebb78c4","Type":"ContainerStarted","Data":"f725efa980554b3b16cc452ee60426af0ded686a6a3d4d420f45e2b246a36780"}
Oct 30 19:33:39.850583 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:39.850288    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:39.967267 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:39.967200    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq" podStartSLOduration=3.042139248 podStartE2EDuration="3.967181287s" totalImagesPullingTime="925.042039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:39.965735468 +0000 UTC m=+3258.608903081" watchObservedRunningTime="2026-10-30 19:33:39.967181287 +0000 UTC m=+3258.610348903"
Oct 30 19:33:44.522791 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:44.522756    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:33:45.876547 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:45.876515    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:33:53.259836 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:33:53.259790    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:33:53.926980 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:53.926944    6014 generic.go:356] "Generic (PLEG): container finished" podID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerID="5886834abaf0027a3dcf3e15e36fe71a9496fc0a807042dcd1213e00ba503ab7" exitCode=0
Oct 30 19:33:53.927300 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:53.927036    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj" event={"ID":"1b3e59b5-20c8-48dc-870f-3382350d32fd","Type":"ContainerDied","Data":"5886834abaf0027a3dcf3e15e36fe71a9496fc0a807042dcd1213e00ba503ab7"}
Oct 30 19:33:53.927581 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:53.927563    6014 scope.go:122] "RemoveContainer" containerID="5886834abaf0027a3dcf3e15e36fe71a9496fc0a807042dcd1213e00ba503ab7"
Oct 30 19:33:53.928855 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:53.928827    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:33:53.928931 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:53.928892    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:33:53.928931 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:53.928922    6014 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:33:54.934178 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:54.934117    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj" event={"ID":"1b3e59b5-20c8-48dc-870f-3382350d32fd","Type":"ContainerStarted","Data":"cbd7461748184f8fae4153201ce47004e6717092901fd1404657f9206c44a882"}
Oct 30 19:33:55.544214 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:55.544172    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:55.545545 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:55.545510    6014 patch_prober.go:29] interesting pod/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 19:33:55.545674 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:55.545562    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj" podUID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 19:33:56.545217 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:56.545181    6014 patch_prober.go:29] interesting pod/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 19:33:56.545645 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:56.545247    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj" podUID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 19:33:57.544554 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:57.544515    6014 patch_prober.go:29] interesting pod/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 19:33:57.544742 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:57.544574    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj" podUID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 19:33:58.545262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:58.545219    6014 patch_prober.go:29] interesting pod/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 19:33:58.545688 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:58.545281    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj" podUID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 19:33:58.780500 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:58.780460    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:33:59.544373 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:59.544333    6014 patch_prober.go:29] interesting pod/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 19:33:59.544566 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:33:59.544402    6014 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj" podUID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 19:34:00.549345 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:00.549316    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:34:00.550532 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:00.550511    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:34:03.528077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:03.528040    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:34:23.266447 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:34:23.266408    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:34:27.829374 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:27.829332    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:34:42.247971 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:42.247882    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:34:53.019638 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.019601    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"]
Oct 30 19:34:53.020105 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.019827    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq" podUID="f049b708-83f0-4e71-8964-f9b49ebb78c4" containerName="maas-api" containerID="cri-o://f725efa980554b3b16cc452ee60426af0ded686a6a3d4d420f45e2b246a36780" gracePeriod=30
Oct 30 19:34:53.265074 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:34:53.265041    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:34:53.513331 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.513299    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:34:53.519424 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.519398    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-maas-api-tls\") pod \"f049b708-83f0-4e71-8964-f9b49ebb78c4\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") "
Oct 30 19:34:53.519569 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.519444    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-metrics-certs\") pod \"f049b708-83f0-4e71-8964-f9b49ebb78c4\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") "
Oct 30 19:34:53.519569 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.519492    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f049b708-83f0-4e71-8964-f9b49ebb78c4-kube-api-access-dhx55\" (UniqueName: \"kubernetes.io/projected/f049b708-83f0-4e71-8964-f9b49ebb78c4-kube-api-access-dhx55\") pod \"f049b708-83f0-4e71-8964-f9b49ebb78c4\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") "
Oct 30 19:34:53.519569 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.519547    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f049b708-83f0-4e71-8964-f9b49ebb78c4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f049b708-83f0-4e71-8964-f9b49ebb78c4-openshift-service-ca\") pod \"f049b708-83f0-4e71-8964-f9b49ebb78c4\" (UID: \"f049b708-83f0-4e71-8964-f9b49ebb78c4\") "
Oct 30 19:34:53.519942 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.519913    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f049b708-83f0-4e71-8964-f9b49ebb78c4-openshift-service-ca" pod "f049b708-83f0-4e71-8964-f9b49ebb78c4" (UID: "f049b708-83f0-4e71-8964-f9b49ebb78c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:53.521746 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.521711    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-maas-api-tls" pod "f049b708-83f0-4e71-8964-f9b49ebb78c4" (UID: "f049b708-83f0-4e71-8964-f9b49ebb78c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:53.521874 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.521831    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f049b708-83f0-4e71-8964-f9b49ebb78c4-kube-api-access-dhx55" pod "f049b708-83f0-4e71-8964-f9b49ebb78c4" (UID: "f049b708-83f0-4e71-8964-f9b49ebb78c4"). InnerVolumeSpecName "kube-api-access-dhx55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:53.521874 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.521853    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-metrics-certs" pod "f049b708-83f0-4e71-8964-f9b49ebb78c4" (UID: "f049b708-83f0-4e71-8964-f9b49ebb78c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:53.620956 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.620917    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:34:53.620956 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.620949    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f049b708-83f0-4e71-8964-f9b49ebb78c4-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:34:53.620956 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.620961    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhx55\" (UniqueName: \"kubernetes.io/projected/f049b708-83f0-4e71-8964-f9b49ebb78c4-kube-api-access-dhx55\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:34:53.621257 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:53.620971    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f049b708-83f0-4e71-8964-f9b49ebb78c4-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:34:54.266587 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:54.266556    6014 generic.go:356] "Generic (PLEG): container finished" podID="f049b708-83f0-4e71-8964-f9b49ebb78c4" containerID="f725efa980554b3b16cc452ee60426af0ded686a6a3d4d420f45e2b246a36780" exitCode=0
Oct 30 19:34:54.267272 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:54.266613    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"
Oct 30 19:34:54.267272 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:54.266642    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq" event={"ID":"f049b708-83f0-4e71-8964-f9b49ebb78c4","Type":"ContainerDied","Data":"f725efa980554b3b16cc452ee60426af0ded686a6a3d4d420f45e2b246a36780"}
Oct 30 19:34:54.267272 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:54.266691    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq" event={"ID":"f049b708-83f0-4e71-8964-f9b49ebb78c4","Type":"ContainerDied","Data":"0faa1f0db8453e537f306c9e12631ed4d83b385d14ea8911e7e4df7f06c416fa"}
Oct 30 19:34:54.267272 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:54.266713    6014 scope.go:122] "RemoveContainer" containerID="f725efa980554b3b16cc452ee60426af0ded686a6a3d4d420f45e2b246a36780"
Oct 30 19:34:54.277335 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:54.277308    6014 scope.go:122] "RemoveContainer" containerID="f725efa980554b3b16cc452ee60426af0ded686a6a3d4d420f45e2b246a36780"
Oct 30 19:34:54.277624 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:34:54.277607    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f725efa980554b3b16cc452ee60426af0ded686a6a3d4d420f45e2b246a36780\": container with ID starting with f725efa980554b3b16cc452ee60426af0ded686a6a3d4d420f45e2b246a36780 not found: ID does not exist" containerID="f725efa980554b3b16cc452ee60426af0ded686a6a3d4d420f45e2b246a36780"
Oct 30 19:34:54.277684 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:54.277634    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f725efa980554b3b16cc452ee60426af0ded686a6a3d4d420f45e2b246a36780"} err="failed to get container status \"f725efa980554b3b16cc452ee60426af0ded686a6a3d4d420f45e2b246a36780\": rpc error: code = NotFound desc = could not find container \"f725efa980554b3b16cc452ee60426af0ded686a6a3d4d420f45e2b246a36780\": container with ID starting with f725efa980554b3b16cc452ee60426af0ded686a6a3d4d420f45e2b246a36780 not found: ID does not exist"
Oct 30 19:34:54.433023 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:54.432991    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"]
Oct 30 19:34:54.434352 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:54.434332    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-682a87-697b75b88c-nlgmq"]
Oct 30 19:34:55.478535 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:55.478504    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f049b708-83f0-4e71-8964-f9b49ebb78c4" path="/var/lib/kubelet/pods/f049b708-83f0-4e71-8964-f9b49ebb78c4/volumes"
Oct 30 19:34:57.549513 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:34:57.549474    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:35:03.977316 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:03.977232    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"]
Oct 30 19:35:03.977771 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:03.977488    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj" podUID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerName="istio-proxy" containerID="cri-o://cbd7461748184f8fae4153201ce47004e6717092901fd1404657f9206c44a882" gracePeriod=30
Oct 30 19:35:09.238262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.238233    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:35:09.344115 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.344085    6014 generic.go:356] "Generic (PLEG): container finished" podID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerID="cbd7461748184f8fae4153201ce47004e6717092901fd1404657f9206c44a882" exitCode=0
Oct 30 19:35:09.344361 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.344170    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"
Oct 30 19:35:09.344361 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.344174    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj" event={"ID":"1b3e59b5-20c8-48dc-870f-3382350d32fd","Type":"ContainerDied","Data":"cbd7461748184f8fae4153201ce47004e6717092901fd1404657f9206c44a882"}
Oct 30 19:35:09.344361 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.344246    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj" event={"ID":"1b3e59b5-20c8-48dc-870f-3382350d32fd","Type":"ContainerDied","Data":"5ba233df6bc2c0ce04d6b13d2f41e36664a940c2dde3a5bf82456c8d94124f42"}
Oct 30 19:35:09.344361 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.344277    6014 scope.go:122] "RemoveContainer" containerID="cbd7461748184f8fae4153201ce47004e6717092901fd1404657f9206c44a882"
Oct 30 19:35:09.353379 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.353360    6014 scope.go:122] "RemoveContainer" containerID="5886834abaf0027a3dcf3e15e36fe71a9496fc0a807042dcd1213e00ba503ab7"
Oct 30 19:35:09.362107 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.362089    6014 scope.go:122] "RemoveContainer" containerID="cbd7461748184f8fae4153201ce47004e6717092901fd1404657f9206c44a882"
Oct 30 19:35:09.362423 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:35:09.362396    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd7461748184f8fae4153201ce47004e6717092901fd1404657f9206c44a882\": container with ID starting with cbd7461748184f8fae4153201ce47004e6717092901fd1404657f9206c44a882 not found: ID does not exist" containerID="cbd7461748184f8fae4153201ce47004e6717092901fd1404657f9206c44a882"
Oct 30 19:35:09.362524 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.362438    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd7461748184f8fae4153201ce47004e6717092901fd1404657f9206c44a882"} err="failed to get container status \"cbd7461748184f8fae4153201ce47004e6717092901fd1404657f9206c44a882\": rpc error: code = NotFound desc = could not find container \"cbd7461748184f8fae4153201ce47004e6717092901fd1404657f9206c44a882\": container with ID starting with cbd7461748184f8fae4153201ce47004e6717092901fd1404657f9206c44a882 not found: ID does not exist"
Oct 30 19:35:09.362524 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.362463    6014 scope.go:122] "RemoveContainer" containerID="5886834abaf0027a3dcf3e15e36fe71a9496fc0a807042dcd1213e00ba503ab7"
Oct 30 19:35:09.362754 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:35:09.362734    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5886834abaf0027a3dcf3e15e36fe71a9496fc0a807042dcd1213e00ba503ab7\": container with ID starting with 5886834abaf0027a3dcf3e15e36fe71a9496fc0a807042dcd1213e00ba503ab7 not found: ID does not exist" containerID="5886834abaf0027a3dcf3e15e36fe71a9496fc0a807042dcd1213e00ba503ab7"
Oct 30 19:35:09.362812 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.362762    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5886834abaf0027a3dcf3e15e36fe71a9496fc0a807042dcd1213e00ba503ab7"} err="failed to get container status \"5886834abaf0027a3dcf3e15e36fe71a9496fc0a807042dcd1213e00ba503ab7\": rpc error: code = NotFound desc = could not find container \"5886834abaf0027a3dcf3e15e36fe71a9496fc0a807042dcd1213e00ba503ab7\": container with ID starting with 5886834abaf0027a3dcf3e15e36fe71a9496fc0a807042dcd1213e00ba503ab7 not found: ID does not exist"
Oct 30 19:35:09.403010 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.402972    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-data\") pod \"1b3e59b5-20c8-48dc-870f-3382350d32fd\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") "
Oct 30 19:35:09.403247 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403069    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3e59b5-20c8-48dc-870f-3382350d32fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3e59b5-20c8-48dc-870f-3382350d32fd-istiod-ca-cert\") pod \"1b3e59b5-20c8-48dc-870f-3382350d32fd\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") "
Oct 30 19:35:09.403247 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403101    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-podinfo\") pod \"1b3e59b5-20c8-48dc-870f-3382350d32fd\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") "
Oct 30 19:35:09.403247 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403133    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-envoy\") pod \"1b3e59b5-20c8-48dc-870f-3382350d32fd\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") "
Oct 30 19:35:09.403446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403256    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-socket\") pod \"1b3e59b5-20c8-48dc-870f-3382350d32fd\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") "
Oct 30 19:35:09.403446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403292    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-kube-api-access-ps865\" (UniqueName: \"kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-kube-api-access-ps865\") pod \"1b3e59b5-20c8-48dc-870f-3382350d32fd\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") "
Oct 30 19:35:09.403446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403298    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-data" pod "1b3e59b5-20c8-48dc-870f-3382350d32fd" (UID: "1b3e59b5-20c8-48dc-870f-3382350d32fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:09.403446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403341    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-certs\") pod \"1b3e59b5-20c8-48dc-870f-3382350d32fd\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") "
Oct 30 19:35:09.403446 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403413    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-credential-socket\") pod \"1b3e59b5-20c8-48dc-870f-3382350d32fd\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") "
Oct 30 19:35:09.403708 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403461    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-token\" (UniqueName: \"kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-token\") pod \"1b3e59b5-20c8-48dc-870f-3382350d32fd\" (UID: \"1b3e59b5-20c8-48dc-870f-3382350d32fd\") "
Oct 30 19:35:09.403708 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403553    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-socket" pod "1b3e59b5-20c8-48dc-870f-3382350d32fd" (UID: "1b3e59b5-20c8-48dc-870f-3382350d32fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:09.403708 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403582    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3e59b5-20c8-48dc-870f-3382350d32fd-istiod-ca-cert" pod "1b3e59b5-20c8-48dc-870f-3382350d32fd" (UID: "1b3e59b5-20c8-48dc-870f-3382350d32fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:09.403708 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403664    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-certs" pod "1b3e59b5-20c8-48dc-870f-3382350d32fd" (UID: "1b3e59b5-20c8-48dc-870f-3382350d32fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:09.403949 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403746    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-credential-socket" pod "1b3e59b5-20c8-48dc-870f-3382350d32fd" (UID: "1b3e59b5-20c8-48dc-870f-3382350d32fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:09.403949 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403937    6014 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:09.404063 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403951    6014 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-workload-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:09.404063 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403964    6014 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-credential-socket\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:09.404063 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403974    6014 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-data\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:09.404063 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.403986    6014 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3e59b5-20c8-48dc-870f-3382350d32fd-istiod-ca-cert\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:09.406134 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.406107    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-podinfo" pod "1b3e59b5-20c8-48dc-870f-3382350d32fd" (UID: "1b3e59b5-20c8-48dc-870f-3382350d32fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:09.406462 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.406429    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-token" pod "1b3e59b5-20c8-48dc-870f-3382350d32fd" (UID: "1b3e59b5-20c8-48dc-870f-3382350d32fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:09.406906 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.406873    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-kube-api-access-ps865" pod "1b3e59b5-20c8-48dc-870f-3382350d32fd" (UID: "1b3e59b5-20c8-48dc-870f-3382350d32fd"). InnerVolumeSpecName "kube-api-access-ps865". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:09.407262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.407238    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-envoy" pod "1b3e59b5-20c8-48dc-870f-3382350d32fd" (UID: "1b3e59b5-20c8-48dc-870f-3382350d32fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:09.506045 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.506011    6014 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-token\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:09.506045 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.506041    6014 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-podinfo\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:09.506045 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.506050    6014 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3e59b5-20c8-48dc-870f-3382350d32fd-istio-envoy\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:09.506286 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.506060    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps865\" (UniqueName: \"kubernetes.io/projected/1b3e59b5-20c8-48dc-870f-3382350d32fd-kube-api-access-ps865\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:09.764093 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.764003    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"]
Oct 30 19:35:09.770994 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:09.770965    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-682a87-openshift-default-84d6f8f57cdn7dj"]
Oct 30 19:35:11.208339 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.208302    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"]
Oct 30 19:35:11.209014 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.208991    6014 cpu_manager.go:414] "containerMap: removing container" podUID="f049b708-83f0-4e71-8964-f9b49ebb78c4" containerName="maas-api"
Oct 30 19:35:11.209014 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.209015    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f049b708-83f0-4e71-8964-f9b49ebb78c4" containerName="maas-api"
Oct 30 19:35:11.209245 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.209036    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerName="istio-proxy"
Oct 30 19:35:11.209245 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.209045    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerName="istio-proxy"
Oct 30 19:35:11.209245 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.209091    6014 cpu_manager.go:414] "containerMap: removing container" podUID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerName="istio-proxy"
Oct 30 19:35:11.209245 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.209099    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerName="istio-proxy"
Oct 30 19:35:11.209451 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.209248    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerName="istio-proxy"
Oct 30 19:35:11.209451 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.209267    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="f049b708-83f0-4e71-8964-f9b49ebb78c4" containerName="maas-api"
Oct 30 19:35:11.209451 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.209282    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3e59b5-20c8-48dc-870f-3382350d32fd" containerName="istio-proxy"
Oct 30 19:35:11.213854 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.213833    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.317440 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.317409    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-00230e87-kserve-self-signed-certs\""
Oct 30 19:35:11.319128 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.319097    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"]
Oct 30 19:35:11.326194 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.326172    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkkzt\" (UniqueName: \"kubernetes.io/projected/5af0dc8d-837d-4239-88b9-77f0d2a22cad-kube-api-access-jkkzt\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.326268 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.326212    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-model-cache\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.326372 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.326351    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tmp-dir\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.326409 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.326398    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-dshm\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.326504 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.326491    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-home\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.326593 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.326581    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tls-certs\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.428068 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.428033    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-dshm\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.428321 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.428095    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-home\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.428321 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.428205    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tls-certs\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.428321 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.428242    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkzt\" (UniqueName: \"kubernetes.io/projected/5af0dc8d-837d-4239-88b9-77f0d2a22cad-kube-api-access-jkkzt\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.428321 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.428274    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-model-cache\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.428559 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.428359    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tmp-dir\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.428635 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.428608    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-home\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.428686 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.428645    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-model-cache\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.428731 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.428689    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tmp-dir\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.430456 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.430428    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-dshm\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.430790 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.430774    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tls-certs\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.475716 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.475637    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3e59b5-20c8-48dc-870f-3382350d32fd" path="/var/lib/kubelet/pods/1b3e59b5-20c8-48dc-870f-3382350d32fd/volumes"
Oct 30 19:35:11.494636 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.494608    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkkzt\" (UniqueName: \"kubernetes.io/projected/5af0dc8d-837d-4239-88b9-77f0d2a22cad-kube-api-access-jkkzt\") pod \"e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") " pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.527900 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.527871    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:11.912671 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:11.912645    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"]
Oct 30 19:35:11.915490 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:35:11.915461    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5af0dc8d_837d_4239_88b9_77f0d2a22cad.slice/crio-ee4f3a9b917df5f9f4e71e1dc23ca201228260666bb69050179ee85da91715b7 WatchSource:0}: Error finding container ee4f3a9b917df5f9f4e71e1dc23ca201228260666bb69050179ee85da91715b7: Status 404 returned error can't find the container with id ee4f3a9b917df5f9f4e71e1dc23ca201228260666bb69050179ee85da91715b7
Oct 30 19:35:12.367782 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:12.367744    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc" event={"ID":"5af0dc8d-837d-4239-88b9-77f0d2a22cad","Type":"ContainerStarted","Data":"cc3151f105bfbba41e0871f1191673037df009ce109f5b5b6a7d030be3cdef80"}
Oct 30 19:35:12.367782 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:12.367784    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc" event={"ID":"5af0dc8d-837d-4239-88b9-77f0d2a22cad","Type":"ContainerStarted","Data":"ee4f3a9b917df5f9f4e71e1dc23ca201228260666bb69050179ee85da91715b7"}
Oct 30 19:35:12.482552 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:12.482483    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc" podStartSLOduration=1.482432491 podStartE2EDuration="1.482432491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:12.481122857 +0000 UTC m=+3351.124290485" watchObservedRunningTime="2026-10-30 19:35:12.482432491 +0000 UTC m=+3351.125600104"
Oct 30 19:35:12.528954 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:12.528920    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:15.651960 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:15.651913    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:35:18.167386 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:18.167295    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5s95d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:35:18.168711 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:18.168081    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5s95d" podUID="c357435d-3e86-4482-8419-a4050103c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:35:21.528779 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:21.528742    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:21.541761 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:21.541734    6014 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:22.458709 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:22.458680    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:35:23.260538 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:35:23.260496    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:35:53.256999 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:35:53.256960    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:35:58.864639 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:35:58.864602    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:36:18.449011 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:18.448971    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:36:21.192489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:21.192439    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9r6z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:36:21.202849 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:21.202795    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9r6z2" podUID="b8882aef-7227-4781-8645-ed9c06928737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:23.258212 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:36:23.258178    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:36:30.404495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:30.404461    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"]
Oct 30 19:36:33.931855 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:33.931816    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:36:45.425000 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.424927    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc" podUID="5af0dc8d-837d-4239-88b9-77f0d2a22cad" containerName="main" containerID="cri-o://cc3151f105bfbba41e0871f1191673037df009ce109f5b5b6a7d030be3cdef80" gracePeriod=45
Oct 30 19:36:45.445793 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.445757    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc" podUID="5af0dc8d-837d-4239-88b9-77f0d2a22cad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.186:8000/ready\": dial tcp 10.134.0.186:8000: connect: connection refused"
Oct 30 19:36:45.708939 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.708903    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:36:45.755916 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.755879    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tmp-dir\") pod \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") "
Oct 30 19:36:45.756091 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.755923    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af0dc8d-837d-4239-88b9-77f0d2a22cad-kube-api-access-jkkzt\" (UniqueName: \"kubernetes.io/projected/5af0dc8d-837d-4239-88b9-77f0d2a22cad-kube-api-access-jkkzt\") pod \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") "
Oct 30 19:36:45.756091 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.755960    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-home\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-home\") pod \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") "
Oct 30 19:36:45.756213 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.756099    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-dshm\") pod \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") "
Oct 30 19:36:45.756213 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.756171    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tls-certs\" (UniqueName: \"kubernetes.io/secret/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tls-certs\") pod \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") "
Oct 30 19:36:45.756305 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.756225    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-model-cache\") pod \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\" (UID: \"5af0dc8d-837d-4239-88b9-77f0d2a22cad\") "
Oct 30 19:36:45.756305 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.756237    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tmp-dir" pod "5af0dc8d-837d-4239-88b9-77f0d2a22cad" (UID: "5af0dc8d-837d-4239-88b9-77f0d2a22cad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:45.756305 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.756283    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-home" pod "5af0dc8d-837d-4239-88b9-77f0d2a22cad" (UID: "5af0dc8d-837d-4239-88b9-77f0d2a22cad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:45.756507 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.756480    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-model-cache" pod "5af0dc8d-837d-4239-88b9-77f0d2a22cad" (UID: "5af0dc8d-837d-4239-88b9-77f0d2a22cad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:45.756993 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.756971    6014 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tmp-dir\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:45.757057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.756999    6014 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-home\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:45.757057 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.757014    6014 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-model-cache\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:45.758405 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.758385    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-dshm" pod "5af0dc8d-837d-4239-88b9-77f0d2a22cad" (UID: "5af0dc8d-837d-4239-88b9-77f0d2a22cad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:45.758495 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.758445    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tls-certs" pod "5af0dc8d-837d-4239-88b9-77f0d2a22cad" (UID: "5af0dc8d-837d-4239-88b9-77f0d2a22cad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:45.758555 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.758537    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af0dc8d-837d-4239-88b9-77f0d2a22cad-kube-api-access-jkkzt" pod "5af0dc8d-837d-4239-88b9-77f0d2a22cad" (UID: "5af0dc8d-837d-4239-88b9-77f0d2a22cad"). InnerVolumeSpecName "kube-api-access-jkkzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:45.858573 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.858522    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkkzt\" (UniqueName: \"kubernetes.io/projected/5af0dc8d-837d-4239-88b9-77f0d2a22cad-kube-api-access-jkkzt\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:45.858573 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.858566    6014 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af0dc8d-837d-4239-88b9-77f0d2a22cad-dshm\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:45.858573 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.858580    6014 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af0dc8d-837d-4239-88b9-77f0d2a22cad-tls-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:45.885313 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.885278    6014 generic.go:356] "Generic (PLEG): container finished" podID="5af0dc8d-837d-4239-88b9-77f0d2a22cad" containerID="cc3151f105bfbba41e0871f1191673037df009ce109f5b5b6a7d030be3cdef80" exitCode=0
Oct 30 19:36:45.885504 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.885322    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"
Oct 30 19:36:45.885504 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.885335    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc" event={"ID":"5af0dc8d-837d-4239-88b9-77f0d2a22cad","Type":"ContainerDied","Data":"cc3151f105bfbba41e0871f1191673037df009ce109f5b5b6a7d030be3cdef80"}
Oct 30 19:36:45.885504 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.885373    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc" event={"ID":"5af0dc8d-837d-4239-88b9-77f0d2a22cad","Type":"ContainerDied","Data":"ee4f3a9b917df5f9f4e71e1dc23ca201228260666bb69050179ee85da91715b7"}
Oct 30 19:36:45.885504 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.885390    6014 scope.go:122] "RemoveContainer" containerID="cc3151f105bfbba41e0871f1191673037df009ce109f5b5b6a7d030be3cdef80"
Oct 30 19:36:45.895253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.895228    6014 scope.go:122] "RemoveContainer" containerID="cc3151f105bfbba41e0871f1191673037df009ce109f5b5b6a7d030be3cdef80"
Oct 30 19:36:45.895550 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:36:45.895524    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3151f105bfbba41e0871f1191673037df009ce109f5b5b6a7d030be3cdef80\": container with ID starting with cc3151f105bfbba41e0871f1191673037df009ce109f5b5b6a7d030be3cdef80 not found: ID does not exist" containerID="cc3151f105bfbba41e0871f1191673037df009ce109f5b5b6a7d030be3cdef80"
Oct 30 19:36:45.895617 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:45.895567    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3151f105bfbba41e0871f1191673037df009ce109f5b5b6a7d030be3cdef80"} err="failed to get container status \"cc3151f105bfbba41e0871f1191673037df009ce109f5b5b6a7d030be3cdef80\": rpc error: code = NotFound desc = could not find container \"cc3151f105bfbba41e0871f1191673037df009ce109f5b5b6a7d030be3cdef80\": container with ID starting with cc3151f105bfbba41e0871f1191673037df009ce109f5b5b6a7d030be3cdef80 not found: ID does not exist"
Oct 30 19:36:46.050775 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:46.050694    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"]
Oct 30 19:36:46.053531 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:46.053507    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-00230e87-kserve-7879f7b849-dm4cc"]
Oct 30 19:36:47.474755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:47.474721    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af0dc8d-837d-4239-88b9-77f0d2a22cad" path="/var/lib/kubelet/pods/5af0dc8d-837d-4239-88b9-77f0d2a22cad/volumes"
Oct 30 19:36:49.713660 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:36:49.713620    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:36:53.257825 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:36:53.257789    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:37:02.422433 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:02.422391    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:37:09.543922 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:09.543883    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:37:11.886166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:11.885295    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dmh57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:11.886166 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:11.885399    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" podUID="aacb99a2-5355-4e64-86bb-ad338415035f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:23.257538 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:37:23.257503    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:37:32.049498 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:32.049461    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:37:32.909893 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:32.909860    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"]
Oct 30 19:37:32.910395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:32.910371    6014 cpu_manager.go:414] "containerMap: removing container" podUID="5af0dc8d-837d-4239-88b9-77f0d2a22cad" containerName="main"
Oct 30 19:37:32.910395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:32.910386    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af0dc8d-837d-4239-88b9-77f0d2a22cad" containerName="main"
Oct 30 19:37:32.910523 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:32.910465    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af0dc8d-837d-4239-88b9-77f0d2a22cad" containerName="main"
Oct 30 19:37:32.914857 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:32.914834    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:32.925620 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:32.925593    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-ff579"]
Oct 30 19:37:32.929294 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:32.929275    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:32.973565 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:32.973534    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:37:32.973565 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:32.973544    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:37:32.973788 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:32.973657    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9wflp\""
Oct 30 19:37:33.016233 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.016171    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:37:33.016488 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.016353    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 19:37:33.019002 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.018977    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"]
Oct 30 19:37:33.020756 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.020732    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-ff579"]
Oct 30 19:37:33.055989 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.055956    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-maas-api-tls\") pod \"maas-api-5c48877c75-69vzd\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:33.055989 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.055992    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42fd58e3-cff8-41fd-ab28-f903ad5d39c4-openshift-service-ca\") pod \"maas-api-595cc648c-ff579\" (UID: \"42fd58e3-cff8-41fd-ab28-f903ad5d39c4\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:33.056499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.056034    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8dsm\" (UniqueName: \"kubernetes.io/projected/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-kube-api-access-p8dsm\") pod \"maas-api-5c48877c75-69vzd\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:33.056499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.056205    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42fd58e3-cff8-41fd-ab28-f903ad5d39c4-metrics-certs\") pod \"maas-api-595cc648c-ff579\" (UID: \"42fd58e3-cff8-41fd-ab28-f903ad5d39c4\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:33.056499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.056265    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-openshift-service-ca\") pod \"maas-api-5c48877c75-69vzd\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:33.056499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.056318    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4mrs\" (UniqueName: \"kubernetes.io/projected/42fd58e3-cff8-41fd-ab28-f903ad5d39c4-kube-api-access-z4mrs\") pod \"maas-api-595cc648c-ff579\" (UID: \"42fd58e3-cff8-41fd-ab28-f903ad5d39c4\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:33.056499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.056381    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-metrics-certs\") pod \"maas-api-5c48877c75-69vzd\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:33.056499 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.056402    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fd58e3-cff8-41fd-ab28-f903ad5d39c4-maas-api-tls\") pod \"maas-api-595cc648c-ff579\" (UID: \"42fd58e3-cff8-41fd-ab28-f903ad5d39c4\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:33.157043 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.157004    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4mrs\" (UniqueName: \"kubernetes.io/projected/42fd58e3-cff8-41fd-ab28-f903ad5d39c4-kube-api-access-z4mrs\") pod \"maas-api-595cc648c-ff579\" (UID: \"42fd58e3-cff8-41fd-ab28-f903ad5d39c4\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:33.157276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.157079    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-metrics-certs\") pod \"maas-api-5c48877c75-69vzd\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:33.157276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.157110    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fd58e3-cff8-41fd-ab28-f903ad5d39c4-maas-api-tls\") pod \"maas-api-595cc648c-ff579\" (UID: \"42fd58e3-cff8-41fd-ab28-f903ad5d39c4\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:33.157276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.157175    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-maas-api-tls\") pod \"maas-api-5c48877c75-69vzd\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:33.157276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.157206    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42fd58e3-cff8-41fd-ab28-f903ad5d39c4-openshift-service-ca\") pod \"maas-api-595cc648c-ff579\" (UID: \"42fd58e3-cff8-41fd-ab28-f903ad5d39c4\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:33.157276 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.157268    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8dsm\" (UniqueName: \"kubernetes.io/projected/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-kube-api-access-p8dsm\") pod \"maas-api-5c48877c75-69vzd\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:33.157593 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.157296    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42fd58e3-cff8-41fd-ab28-f903ad5d39c4-metrics-certs\") pod \"maas-api-595cc648c-ff579\" (UID: \"42fd58e3-cff8-41fd-ab28-f903ad5d39c4\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:33.157593 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.157365    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-openshift-service-ca\") pod \"maas-api-5c48877c75-69vzd\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:33.158070 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.158043    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-openshift-service-ca\") pod \"maas-api-5c48877c75-69vzd\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:33.158286 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.158261    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42fd58e3-cff8-41fd-ab28-f903ad5d39c4-openshift-service-ca\") pod \"maas-api-595cc648c-ff579\" (UID: \"42fd58e3-cff8-41fd-ab28-f903ad5d39c4\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:33.159949 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.159894    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-maas-api-tls\") pod \"maas-api-5c48877c75-69vzd\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:33.160048 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.159973    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fd58e3-cff8-41fd-ab28-f903ad5d39c4-maas-api-tls\") pod \"maas-api-595cc648c-ff579\" (UID: \"42fd58e3-cff8-41fd-ab28-f903ad5d39c4\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:33.160300 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.160283    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42fd58e3-cff8-41fd-ab28-f903ad5d39c4-metrics-certs\") pod \"maas-api-595cc648c-ff579\" (UID: \"42fd58e3-cff8-41fd-ab28-f903ad5d39c4\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:33.160349 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.160330    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-metrics-certs\") pod \"maas-api-5c48877c75-69vzd\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:33.215302 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.215271    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8dsm\" (UniqueName: \"kubernetes.io/projected/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-kube-api-access-p8dsm\") pod \"maas-api-5c48877c75-69vzd\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:33.215476 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.215458    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4mrs\" (UniqueName: \"kubernetes.io/projected/42fd58e3-cff8-41fd-ab28-f903ad5d39c4-kube-api-access-z4mrs\") pod \"maas-api-595cc648c-ff579\" (UID: \"42fd58e3-cff8-41fd-ab28-f903ad5d39c4\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:33.228881 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.228844    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:33.246237 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.246184    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:33.609965 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.609938    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"]
Oct 30 19:37:33.614728 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:37:33.614656    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5942f4b_b8ea_4db9_aea3_41fd168f45ff.slice/crio-fdefc6e8465101c4d34c260aa16e05de437cfecd8355707c873deac3cc6d067f WatchSource:0}: Error finding container fdefc6e8465101c4d34c260aa16e05de437cfecd8355707c873deac3cc6d067f: Status 404 returned error can't find the container with id fdefc6e8465101c4d34c260aa16e05de437cfecd8355707c873deac3cc6d067f
Oct 30 19:37:33.615734 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.615714    6014 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:37:33.657773 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:33.657747    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-ff579"]
Oct 30 19:37:33.660429 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:37:33.660403    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fd58e3_cff8_41fd_ab28_f903ad5d39c4.slice/crio-650d97f33cf26acf0490e106e422e263bb190c90a286e571c7e1e5c405ba6614 WatchSource:0}: Error finding container 650d97f33cf26acf0490e106e422e263bb190c90a286e571c7e1e5c405ba6614: Status 404 returned error can't find the container with id 650d97f33cf26acf0490e106e422e263bb190c90a286e571c7e1e5c405ba6614
Oct 30 19:37:34.256644 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:34.256603    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd" event={"ID":"b5942f4b-b8ea-4db9-aea3-41fd168f45ff","Type":"ContainerStarted","Data":"fdefc6e8465101c4d34c260aa16e05de437cfecd8355707c873deac3cc6d067f"}
Oct 30 19:37:34.257806 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:34.257779    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579" event={"ID":"42fd58e3-cff8-41fd-ab28-f903ad5d39c4","Type":"ContainerStarted","Data":"650d97f33cf26acf0490e106e422e263bb190c90a286e571c7e1e5c405ba6614"}
Oct 30 19:37:35.265314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:35.265271    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579" event={"ID":"42fd58e3-cff8-41fd-ab28-f903ad5d39c4","Type":"ContainerStarted","Data":"22d38e4d4d7713a590ee98ac7d13e7ec46d40f420a29f22ec1ee3c7343cf3680"}
Oct 30 19:37:35.265825 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:35.265347    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:35.266910 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:35.266885    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd" event={"ID":"b5942f4b-b8ea-4db9-aea3-41fd168f45ff","Type":"ContainerStarted","Data":"fd70f7576b88483d0fe1fb2391d1f052abd5cda5b0734ba750f5a00565402c78"}
Oct 30 19:37:35.267042 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:35.267019    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:35.380762 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:35.380702    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579" podStartSLOduration=2.583782436 podStartE2EDuration="3.38068508s" totalImagesPullingTime="796.902644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:35.379464359 +0000 UTC m=+3494.022631976" watchObservedRunningTime="2026-10-30 19:37:35.38068508 +0000 UTC m=+3494.023852693"
Oct 30 19:37:35.497755 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:35.497696    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd" podStartSLOduration=2.634067722 podStartE2EDuration="3.497677723s" totalImagesPullingTime="863.610001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:35.496556349 +0000 UTC m=+3494.139723976" watchObservedRunningTime="2026-10-30 19:37:35.497677723 +0000 UTC m=+3494.140845383"
Oct 30 19:37:41.277027 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.276992    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:41.277510 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.277415    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-ff579"
Oct 30 19:37:41.544947 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.544769    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"]
Oct 30 19:37:41.545268 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.545201    6014 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd" podUID="b5942f4b-b8ea-4db9-aea3-41fd168f45ff" containerName="maas-api" containerID="cri-o://fd70f7576b88483d0fe1fb2391d1f052abd5cda5b0734ba750f5a00565402c78" gracePeriod=30
Oct 30 19:37:41.797326 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.797262    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:41.952876 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.952848    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-kube-api-access-p8dsm\" (UniqueName: \"kubernetes.io/projected/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-kube-api-access-p8dsm\") pod \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") "
Oct 30 19:37:41.953077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.952980    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-openshift-service-ca\") pod \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") "
Oct 30 19:37:41.953077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.953028    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-maas-api-tls\") pod \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") "
Oct 30 19:37:41.953077 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.953058    6014 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-metrics-certs\") pod \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\" (UID: \"b5942f4b-b8ea-4db9-aea3-41fd168f45ff\") "
Oct 30 19:37:41.953395 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.953368    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-openshift-service-ca" pod "b5942f4b-b8ea-4db9-aea3-41fd168f45ff" (UID: "b5942f4b-b8ea-4db9-aea3-41fd168f45ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:41.953530 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.953468    6014 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-openshift-service-ca\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:41.955330 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.955298    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-maas-api-tls" pod "b5942f4b-b8ea-4db9-aea3-41fd168f45ff" (UID: "b5942f4b-b8ea-4db9-aea3-41fd168f45ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:41.955439 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.955392    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-metrics-certs" pod "b5942f4b-b8ea-4db9-aea3-41fd168f45ff" (UID: "b5942f4b-b8ea-4db9-aea3-41fd168f45ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:41.955439 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:41.955424    6014 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-kube-api-access-p8dsm" pod "b5942f4b-b8ea-4db9-aea3-41fd168f45ff" (UID: "b5942f4b-b8ea-4db9-aea3-41fd168f45ff"). InnerVolumeSpecName "kube-api-access-p8dsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:42.055084 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:42.054982    6014 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-maas-api-tls\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:42.055084 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:42.055018    6014 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-metrics-certs\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:42.055084 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:42.055032    6014 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8dsm\" (UniqueName: \"kubernetes.io/projected/b5942f4b-b8ea-4db9-aea3-41fd168f45ff-kube-api-access-p8dsm\") on node \"ip-10-0-143-97.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:42.303653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:42.303615    6014 generic.go:356] "Generic (PLEG): container finished" podID="b5942f4b-b8ea-4db9-aea3-41fd168f45ff" containerID="fd70f7576b88483d0fe1fb2391d1f052abd5cda5b0734ba750f5a00565402c78" exitCode=0
Oct 30 19:37:42.303653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:42.303665    6014 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"
Oct 30 19:37:42.304236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:42.303687    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd" event={"ID":"b5942f4b-b8ea-4db9-aea3-41fd168f45ff","Type":"ContainerDied","Data":"fd70f7576b88483d0fe1fb2391d1f052abd5cda5b0734ba750f5a00565402c78"}
Oct 30 19:37:42.304236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:42.303740    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-69vzd" event={"ID":"b5942f4b-b8ea-4db9-aea3-41fd168f45ff","Type":"ContainerDied","Data":"fdefc6e8465101c4d34c260aa16e05de437cfecd8355707c873deac3cc6d067f"}
Oct 30 19:37:42.304236 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:42.303771    6014 scope.go:122] "RemoveContainer" containerID="fd70f7576b88483d0fe1fb2391d1f052abd5cda5b0734ba750f5a00565402c78"
Oct 30 19:37:42.313653 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:42.313633    6014 scope.go:122] "RemoveContainer" containerID="fd70f7576b88483d0fe1fb2391d1f052abd5cda5b0734ba750f5a00565402c78"
Oct 30 19:37:42.313973 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:37:42.313954    6014 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd70f7576b88483d0fe1fb2391d1f052abd5cda5b0734ba750f5a00565402c78\": container with ID starting with fd70f7576b88483d0fe1fb2391d1f052abd5cda5b0734ba750f5a00565402c78 not found: ID does not exist" containerID="fd70f7576b88483d0fe1fb2391d1f052abd5cda5b0734ba750f5a00565402c78"
Oct 30 19:37:42.314021 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:42.313989    6014 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd70f7576b88483d0fe1fb2391d1f052abd5cda5b0734ba750f5a00565402c78"} err="failed to get container status \"fd70f7576b88483d0fe1fb2391d1f052abd5cda5b0734ba750f5a00565402c78\": rpc error: code = NotFound desc = could not find container \"fd70f7576b88483d0fe1fb2391d1f052abd5cda5b0734ba750f5a00565402c78\": container with ID starting with fd70f7576b88483d0fe1fb2391d1f052abd5cda5b0734ba750f5a00565402c78 not found: ID does not exist"
Oct 30 19:37:42.478168 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:42.478111    6014 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"]
Oct 30 19:37:42.489793 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:42.489759    6014 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-69vzd"]
Oct 30 19:37:43.359771 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:43.359734    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:37:43.475672 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:43.475634    6014 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5942f4b-b8ea-4db9-aea3-41fd168f45ff" path="/var/lib/kubelet/pods/b5942f4b-b8ea-4db9-aea3-41fd168f45ff/volumes"
Oct 30 19:37:53.257554 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:37:53.257515    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:37:54.257637 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:54.257604    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:37:56.880309 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:56.880263    6014 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dmh57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:56.881010 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:37:56.880975    6014 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dmh57" podUID="aacb99a2-5355-4e64-86bb-ad338415035f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:15.514502 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:38:15.514461    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:38:23.259104 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:38:23.259060    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:38:47.447251 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:38:47.447211    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bdl7l"]
Oct 30 19:38:53.258805 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:38:53.258772    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:38:58.285797 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:38:58.285768    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58bf7d8f44-rrkp2_54639bef-cb05-4ea2-8d95-bc17c5bb6cb9/manager/0.log"
Oct 30 19:38:58.885307 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:38:58.885274    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-cc5rr_f3c033e3-63bc-4373-a7bf-545fbe3a7f40/manager/0.log"
Oct 30 19:38:59.516284 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:38:59.516256    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-994975fdf-hb6nt_4405e9f5-ccac-4971-84f7-24612c1de6c2/manager/0.log"
Oct 30 19:39:00.047022 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:00.046991    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-s42cx_bd39a056-ee8f-40c4-8837-bea0bbd41085/manager/0.log"
Oct 30 19:39:00.298875 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:00.298779    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-nj87z_3794243f-5a1c-4529-ab37-87509cd04491/manager/0.log"
Oct 30 19:39:00.883606 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:00.883525    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cb69c88fb-dnd2f_a0083dba-963f-4e40-a5cf-b53d53563918/copy-manifests/0.log"
Oct 30 19:39:00.960540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:00.960503    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cb69c88fb-dnd2f_a0083dba-963f-4e40-a5cf-b53d53563918/manager/0.log"
Oct 30 19:39:02.361579 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:02.361547    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-ff579_42fd58e3-cff8-41fd-ab28-f903ad5d39c4/maas-api/0.log"
Oct 30 19:39:06.482832 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:06.482779    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-9qpl9_7a1d7873-6426-4c65-bc71-59f5a7ab95cb/manager/0.log"
Oct 30 19:39:07.653310 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:07.653279    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bdl7l_e1d97859-2437-49ca-9dc4-7e5d381b2e86/limitador/0.log"
Oct 30 19:39:08.812016 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:08.811988    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5zffhh_34b518da-36c0-4652-b3bd-e1053a1226a3/istio-proxy/0.log"
Oct 30 19:39:09.777263 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:09.777229    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-5s95d_c357435d-3e86-4482-8419-a4050103c921/istio-proxy/0.log"
Oct 30 19:39:10.783683 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:10.783654    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-9r6z2_b8882aef-7227-4781-8645-ed9c06928737/istio-proxy/0.log"
Oct 30 19:39:11.049243 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:11.049166    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-dmh57_aacb99a2-5355-4e64-86bb-ad338415035f/istio-proxy/0.log"
Oct 30 19:39:12.282724 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:12.282695    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-v2wnm_21ca6136-afbb-43d7-96ac-9b501a95c63e/payload-pre-processing/0.log"
Oct 30 19:39:12.510565 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:12.510541    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2jfvg_5cc63475-ec73-451a-8c47-0dd82495ed48/payload-processing/0.log"
Oct 30 19:39:13.905299 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:13.905258    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-67kkz_2d7fbb25-1611-4efd-83fb-c636fdc232ff/main/0.log"
Oct 30 19:39:14.137234 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:14.137206    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-z6rtg_e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208/main/0.log"
Oct 30 19:39:14.366846 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:14.366811    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-jg64x_79f5eaf5-cfd1-44df-878c-70c7798c2c12/main/0.log"
Oct 30 19:39:14.863361 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:14.863327    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c_5596ab6c-2bae-460e-81c1-e875ce418e88/main/0.log"
Oct 30 19:39:23.258283 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:39:23.258249    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:39:25.577399 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:25.577365    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58bf7d8f44-rrkp2_54639bef-cb05-4ea2-8d95-bc17c5bb6cb9/manager/0.log"
Oct 30 19:39:26.174411 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:26.174380    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-cc5rr_f3c033e3-63bc-4373-a7bf-545fbe3a7f40/manager/0.log"
Oct 30 19:39:26.740828 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:26.740790    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-994975fdf-hb6nt_4405e9f5-ccac-4971-84f7-24612c1de6c2/manager/0.log"
Oct 30 19:39:27.211935 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:27.211900    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-s42cx_bd39a056-ee8f-40c4-8837-bea0bbd41085/manager/0.log"
Oct 30 19:39:27.468407 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:27.468318    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-nj87z_3794243f-5a1c-4529-ab37-87509cd04491/manager/0.log"
Oct 30 19:39:28.068617 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:28.068588    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cb69c88fb-dnd2f_a0083dba-963f-4e40-a5cf-b53d53563918/copy-manifests/0.log"
Oct 30 19:39:28.143990 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:28.143955    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cb69c88fb-dnd2f_a0083dba-963f-4e40-a5cf-b53d53563918/manager/0.log"
Oct 30 19:39:29.322126 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:29.322097    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-ff579_42fd58e3-cff8-41fd-ab28-f903ad5d39c4/maas-api/0.log"
Oct 30 19:39:33.145929 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:33.145841    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-9qpl9_7a1d7873-6426-4c65-bc71-59f5a7ab95cb/manager/0.log"
Oct 30 19:39:34.244017 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:34.243990    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bdl7l_e1d97859-2437-49ca-9dc4-7e5d381b2e86/limitador/0.log"
Oct 30 19:39:35.181019 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:35.180990    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5zffhh_34b518da-36c0-4652-b3bd-e1053a1226a3/istio-proxy/0.log"
Oct 30 19:39:36.136567 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:36.136533    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-5s95d_c357435d-3e86-4482-8419-a4050103c921/istio-proxy/0.log"
Oct 30 19:39:37.099107 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:37.099078    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-9r6z2_b8882aef-7227-4781-8645-ed9c06928737/istio-proxy/0.log"
Oct 30 19:39:37.372072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:37.371986    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-dmh57_aacb99a2-5355-4e64-86bb-ad338415035f/istio-proxy/0.log"
Oct 30 19:39:38.587816 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:38.587780    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-v2wnm_21ca6136-afbb-43d7-96ac-9b501a95c63e/payload-pre-processing/0.log"
Oct 30 19:39:38.807397 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:38.807360    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2jfvg_5cc63475-ec73-451a-8c47-0dd82495ed48/payload-processing/0.log"
Oct 30 19:39:39.863786 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:39.863755    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-67kkz_2d7fbb25-1611-4efd-83fb-c636fdc232ff/main/0.log"
Oct 30 19:39:40.094334 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:40.094303    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-z6rtg_e7694c0f-a6d7-4a3a-bbd8-b2d59bce2208/main/0.log"
Oct 30 19:39:40.334739 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:40.334709    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-jg64x_79f5eaf5-cfd1-44df-878c-70c7798c2c12/main/0.log"
Oct 30 19:39:40.841830 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:39:40.841800    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-rdg2c_5596ab6c-2bae-460e-81c1-e875ce418e88/main/0.log"
Oct 30 19:39:53.257609 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:39:53.257567    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:40:11.043067 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:11.043029    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-994975fdf-hb6nt_4405e9f5-ccac-4971-84f7-24612c1de6c2/manager/0.log"
Oct 30 19:40:11.547031 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:11.546995    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-994975fdf-hb6nt_4405e9f5-ccac-4971-84f7-24612c1de6c2/manager/0.log"
Oct 30 19:40:12.291618 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:12.291591    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2jfvg_5cc63475-ec73-451a-8c47-0dd82495ed48/payload-processing/0.log"
Oct 30 19:40:12.519456 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:12.519418    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2jfvg_5cc63475-ec73-451a-8c47-0dd82495ed48/payload-processing/0.log"
Oct 30 19:40:23.257661 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:40:23.257625    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:40:50.488029 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:50.487992    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-994975fdf-hb6nt_4405e9f5-ccac-4971-84f7-24612c1de6c2/manager/0.log"
Oct 30 19:40:51.065990 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:51.065952    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-994975fdf-hb6nt_4405e9f5-ccac-4971-84f7-24612c1de6c2/manager/0.log"
Oct 30 19:40:51.831695 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:51.831665    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2jfvg_5cc63475-ec73-451a-8c47-0dd82495ed48/payload-processing/0.log"
Oct 30 19:40:52.066761 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.066732    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2jfvg_5cc63475-ec73-451a-8c47-0dd82495ed48/payload-processing/0.log"
Oct 30 19:40:52.441948 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.441903    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vh98j/must-gather-tjcfr"]
Oct 30 19:40:52.442445 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.442429    6014 cpu_manager.go:414] "containerMap: removing container" podUID="b5942f4b-b8ea-4db9-aea3-41fd168f45ff" containerName="maas-api"
Oct 30 19:40:52.442503 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.442446    6014 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5942f4b-b8ea-4db9-aea3-41fd168f45ff" containerName="maas-api"
Oct 30 19:40:52.442540 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.442516    6014 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5942f4b-b8ea-4db9-aea3-41fd168f45ff" containerName="maas-api"
Oct 30 19:40:52.446018 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.446002    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vh98j/must-gather-tjcfr"
Oct 30 19:40:52.498209 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.498176    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vh98j\"/\"openshift-service-ca.crt\""
Oct 30 19:40:52.547634 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.547597    6014 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vh98j\"/\"kube-root-ca.crt\""
Oct 30 19:40:52.547826 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.547694    6014 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vh98j\"/\"default-dockercfg-wmjvf\""
Oct 30 19:40:52.549214 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.549194    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vh98j/must-gather-tjcfr"]
Oct 30 19:40:52.625014 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.624977    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/685526a5-35bd-4f3a-83d6-20f6f7339b58-must-gather-output\") pod \"must-gather-tjcfr\" (UID: \"685526a5-35bd-4f3a-83d6-20f6f7339b58\") " pod="openshift-must-gather-vh98j/must-gather-tjcfr"
Oct 30 19:40:52.625254 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.625210    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98tgw\" (UniqueName: \"kubernetes.io/projected/685526a5-35bd-4f3a-83d6-20f6f7339b58-kube-api-access-98tgw\") pod \"must-gather-tjcfr\" (UID: \"685526a5-35bd-4f3a-83d6-20f6f7339b58\") " pod="openshift-must-gather-vh98j/must-gather-tjcfr"
Oct 30 19:40:52.726226 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.726107    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/685526a5-35bd-4f3a-83d6-20f6f7339b58-must-gather-output\") pod \"must-gather-tjcfr\" (UID: \"685526a5-35bd-4f3a-83d6-20f6f7339b58\") " pod="openshift-must-gather-vh98j/must-gather-tjcfr"
Oct 30 19:40:52.726418 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.726237    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98tgw\" (UniqueName: \"kubernetes.io/projected/685526a5-35bd-4f3a-83d6-20f6f7339b58-kube-api-access-98tgw\") pod \"must-gather-tjcfr\" (UID: \"685526a5-35bd-4f3a-83d6-20f6f7339b58\") " pod="openshift-must-gather-vh98j/must-gather-tjcfr"
Oct 30 19:40:52.726489 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.726426    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/685526a5-35bd-4f3a-83d6-20f6f7339b58-must-gather-output\") pod \"must-gather-tjcfr\" (UID: \"685526a5-35bd-4f3a-83d6-20f6f7339b58\") " pod="openshift-must-gather-vh98j/must-gather-tjcfr"
Oct 30 19:40:52.783661 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:52.783631    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98tgw\" (UniqueName: \"kubernetes.io/projected/685526a5-35bd-4f3a-83d6-20f6f7339b58-kube-api-access-98tgw\") pod \"must-gather-tjcfr\" (UID: \"685526a5-35bd-4f3a-83d6-20f6f7339b58\") " pod="openshift-must-gather-vh98j/must-gather-tjcfr"
Oct 30 19:40:53.060669 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:53.060579    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vh98j/must-gather-tjcfr"
Oct 30 19:40:53.257316 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:40:53.257274    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:40:53.433262 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:53.433051    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vh98j/must-gather-tjcfr"]
Oct 30 19:40:53.436306 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:40:53.436279    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod685526a5_35bd_4f3a_83d6_20f6f7339b58.slice/crio-74cf88de07febed58e724d81fccbc43eeb2a16638af27994036264dbf23d6b71 WatchSource:0}: Error finding container 74cf88de07febed58e724d81fccbc43eeb2a16638af27994036264dbf23d6b71: Status 404 returned error can't find the container with id 74cf88de07febed58e724d81fccbc43eeb2a16638af27994036264dbf23d6b71
Oct 30 19:40:54.295336 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:54.295302    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vh98j/must-gather-tjcfr" event={"ID":"685526a5-35bd-4f3a-83d6-20f6f7339b58","Type":"ContainerStarted","Data":"74cf88de07febed58e724d81fccbc43eeb2a16638af27994036264dbf23d6b71"}
Oct 30 19:40:55.302111 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:55.302020    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vh98j/must-gather-tjcfr" event={"ID":"685526a5-35bd-4f3a-83d6-20f6f7339b58","Type":"ContainerStarted","Data":"d91cb52f7c7f069975e16ca5044c7a081f2e241e472bae90a3ed9efee3f934c7"}
Oct 30 19:40:55.302111 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:55.302077    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vh98j/must-gather-tjcfr" event={"ID":"685526a5-35bd-4f3a-83d6-20f6f7339b58","Type":"ContainerStarted","Data":"1079af9a205f0803883e7a10d003743219198bf2031addabc020b59707bc91da"}
Oct 30 19:40:55.416178 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:40:55.416105    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vh98j/must-gather-tjcfr" podStartSLOduration=2.08494898 podStartE2EDuration="3.416089688s" totalImagesPullingTime="1.331140708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:55.414352699 +0000 UTC m=+3694.057520312" watchObservedRunningTime="2026-10-30 19:40:55.416089688 +0000 UTC m=+3694.059257302"
Oct 30 19:41:01.910229 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:01.903290    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7h2cm_ab2168bb-3bf1-48ed-993d-4d0a014f00ae/global-pull-secret-syncer/0.log"
Oct 30 19:41:05.955353 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:05.955318    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wwvr8_4d70a762-bdea-4394-8a95-5e5f0fb4042c/konnectivity-agent/0.log"
Oct 30 19:41:07.207708 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:07.207664    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-97.us-west-1.compute.internal_5a238862e2cf33c9dc387d53f60c786c/haproxy/0.log"
Oct 30 19:41:11.587601 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:11.587552    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-9qpl9_7a1d7873-6426-4c65-bc71-59f5a7ab95cb/manager/0.log"
Oct 30 19:41:11.971001 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:11.970923    6014 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"]
Oct 30 19:41:11.978224 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:11.978200    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.037215 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.037177    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"]
Oct 30 19:41:12.165097 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.165058    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ffc1176e-4743-414b-9220-4cd424d04e15-podres\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.165314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.165108    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ffc1176e-4743-414b-9220-4cd424d04e15-proc\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.165314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.165130    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffc1176e-4743-414b-9220-4cd424d04e15-sys\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.165314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.165233    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffc1176e-4743-414b-9220-4cd424d04e15-lib-modules\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.165314 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.165281    6014 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7t54\" (UniqueName: \"kubernetes.io/projected/ffc1176e-4743-414b-9220-4cd424d04e15-kube-api-access-z7t54\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.266966 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.266877    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ffc1176e-4743-414b-9220-4cd424d04e15-podres\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.266966 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.266937    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ffc1176e-4743-414b-9220-4cd424d04e15-proc\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.266966 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.266964    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffc1176e-4743-414b-9220-4cd424d04e15-sys\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.267253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.266988    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffc1176e-4743-414b-9220-4cd424d04e15-lib-modules\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.267253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.267007    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ffc1176e-4743-414b-9220-4cd424d04e15-podres\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.267253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.267014    6014 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7t54\" (UniqueName: \"kubernetes.io/projected/ffc1176e-4743-414b-9220-4cd424d04e15-kube-api-access-z7t54\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.267253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.267070    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffc1176e-4743-414b-9220-4cd424d04e15-sys\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.267253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.267072    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ffc1176e-4743-414b-9220-4cd424d04e15-proc\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.267253 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.267094    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffc1176e-4743-414b-9220-4cd424d04e15-lib-modules\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.324521 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.324484    6014 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7t54\" (UniqueName: \"kubernetes.io/projected/ffc1176e-4743-414b-9220-4cd424d04e15-kube-api-access-z7t54\") pod \"perf-node-gather-daemonset-z8xxr\" (UID: \"ffc1176e-4743-414b-9220-4cd424d04e15\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.592959 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.592927    6014 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:12.975831 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:12.975783    6014 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"]
Oct 30 19:41:12.978727 ip-10-0-143-97 kubenswrapper[6014]: W1030 19:41:12.978682    6014 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podffc1176e_4743_414b_9220_4cd424d04e15.slice/crio-a779059e174ce9d91ea41989eb2bf00e4928e572c1668677ec9d5b0cb3a0e2a4 WatchSource:0}: Error finding container a779059e174ce9d91ea41989eb2bf00e4928e572c1668677ec9d5b0cb3a0e2a4: Status 404 returned error can't find the container with id a779059e174ce9d91ea41989eb2bf00e4928e572c1668677ec9d5b0cb3a0e2a4
Oct 30 19:41:13.280082 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:13.279999    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bdl7l_e1d97859-2437-49ca-9dc4-7e5d381b2e86/limitador/0.log"
Oct 30 19:41:13.437875 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:13.437837    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr" event={"ID":"ffc1176e-4743-414b-9220-4cd424d04e15","Type":"ContainerStarted","Data":"0c4452440c2eff156bc7e8e9d9a1439ea4e55fd26e27fcd37fe36a6b1bdee71e"}
Oct 30 19:41:13.438072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:13.437908    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:13.438072 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:13.437934    6014 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr" event={"ID":"ffc1176e-4743-414b-9220-4cd424d04e15","Type":"ContainerStarted","Data":"a779059e174ce9d91ea41989eb2bf00e4928e572c1668677ec9d5b0cb3a0e2a4"}
Oct 30 19:41:13.556762 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:13.556652    6014 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr" podStartSLOduration=2.556637895 podStartE2EDuration="2.556637895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:13.554554497 +0000 UTC m=+3712.197722111" watchObservedRunningTime="2026-10-30 19:41:13.556637895 +0000 UTC m=+3712.199805506"
Oct 30 19:41:16.384090 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:16.384055    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1156af79-a794-494b-98aa-28f8f0f42e58/alertmanager/0.log"
Oct 30 19:41:16.798400 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:16.798314    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1156af79-a794-494b-98aa-28f8f0f42e58/config-reloader/0.log"
Oct 30 19:41:17.232444 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:17.232410    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1156af79-a794-494b-98aa-28f8f0f42e58/kube-rbac-proxy-web/0.log"
Oct 30 19:41:17.655921 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:17.655887    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1156af79-a794-494b-98aa-28f8f0f42e58/kube-rbac-proxy/0.log"
Oct 30 19:41:18.067901 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:18.067874    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1156af79-a794-494b-98aa-28f8f0f42e58/kube-rbac-proxy-metric/0.log"
Oct 30 19:41:18.485358 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:18.485286    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1156af79-a794-494b-98aa-28f8f0f42e58/prom-label-proxy/0.log"
Oct 30 19:41:18.903036 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:18.903007    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1156af79-a794-494b-98aa-28f8f0f42e58/init-config-reloader/0.log"
Oct 30 19:41:19.455595 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:19.455564    6014 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-z8xxr"
Oct 30 19:41:19.754468 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:19.754360    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2fxg9_a1ed51f8-138a-4e90-bcfe-00cae5eba235/kube-state-metrics/0.log"
Oct 30 19:41:20.161547 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:20.161521    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2fxg9_a1ed51f8-138a-4e90-bcfe-00cae5eba235/kube-rbac-proxy-main/0.log"
Oct 30 19:41:20.576415 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:20.576386    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2fxg9_a1ed51f8-138a-4e90-bcfe-00cae5eba235/kube-rbac-proxy-self/0.log"
Oct 30 19:41:21.011308 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:21.011235    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7848bc9d85-wmsb2_6cebbc4c-d198-4222-968d-4ab6d035ee51/metrics-server/0.log"
Oct 30 19:41:21.441553 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:21.441520    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-gbbdc_909bd6eb-4140-4939-a5c1-4d148b3791dc/monitoring-plugin/0.log"
Oct 30 19:41:21.863090 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:21.863045    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h2x92_5d438d4b-2388-4a65-a199-8cb4b3d27dc2/node-exporter/0.log"
Oct 30 19:41:22.280855 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:22.280771    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h2x92_5d438d4b-2388-4a65-a199-8cb4b3d27dc2/kube-rbac-proxy/0.log"
Oct 30 19:41:22.694656 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:22.694632    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h2x92_5d438d4b-2388-4a65-a199-8cb4b3d27dc2/init-textfile/0.log"
Oct 30 19:41:23.263519 ip-10-0-143-97 kubenswrapper[6014]: E1030 19:41:23.263478    6014 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fzvff" containerName="extractor"
Oct 30 19:41:25.633725 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:25.633692    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jbhtt_717a2238-075b-4288-9c74-1a306efb331d/kube-rbac-proxy-main/0.log"
Oct 30 19:41:26.050202 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:26.050119    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jbhtt_717a2238-075b-4288-9c74-1a306efb331d/kube-rbac-proxy-self/0.log"
Oct 30 19:41:26.464939 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:26.464889    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jbhtt_717a2238-075b-4288-9c74-1a306efb331d/openshift-state-metrics/0.log"
Oct 30 19:41:26.950458 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:26.950419    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff/prometheus/0.log"
Oct 30 19:41:27.371384 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:27.371357    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff/config-reloader/0.log"
Oct 30 19:41:27.785837 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:27.785756    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff/thanos-sidecar/0.log"
Oct 30 19:41:28.199970 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:28.199944    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff/kube-rbac-proxy-web/0.log"
Oct 30 19:41:28.614965 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:28.614940    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff/kube-rbac-proxy/0.log"
Oct 30 19:41:29.031161 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:29.031060    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff/kube-rbac-proxy-thanos/0.log"
Oct 30 19:41:29.448023 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:29.447991    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f80fc66-0bc4-48c7-9f2f-2d63df3c5dff/init-config-reloader/0.log"
Oct 30 19:41:31.134859 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:31.134826    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78cd67666d-td5fr_2cf0ddaf-aa3c-4390-b911-cab27953af11/telemeter-client/0.log"
Oct 30 19:41:31.550132 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:31.550065    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78cd67666d-td5fr_2cf0ddaf-aa3c-4390-b911-cab27953af11/reload/0.log"
Oct 30 19:41:31.964794 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:31.964759    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78cd67666d-td5fr_2cf0ddaf-aa3c-4390-b911-cab27953af11/kube-rbac-proxy/0.log"
Oct 30 19:41:32.386643 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:32.386612    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68b7f4fd8f-mttl7_a6cfb91c-11df-4d59-b723-320754d81b99/thanos-query/0.log"
Oct 30 19:41:32.798866 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:32.798797    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68b7f4fd8f-mttl7_a6cfb91c-11df-4d59-b723-320754d81b99/kube-rbac-proxy-web/0.log"
Oct 30 19:41:33.211416 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:33.211372    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68b7f4fd8f-mttl7_a6cfb91c-11df-4d59-b723-320754d81b99/kube-rbac-proxy/0.log"
Oct 30 19:41:33.625124 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:33.625089    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68b7f4fd8f-mttl7_a6cfb91c-11df-4d59-b723-320754d81b99/prom-label-proxy/0.log"
Oct 30 19:41:34.039238 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:34.039164    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68b7f4fd8f-mttl7_a6cfb91c-11df-4d59-b723-320754d81b99/kube-rbac-proxy-rules/0.log"
Oct 30 19:41:34.454255 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:34.454218    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68b7f4fd8f-mttl7_a6cfb91c-11df-4d59-b723-320754d81b99/kube-rbac-proxy-metrics/0.log"
Oct 30 19:41:34.795467 ip-10-0-143-97 kubenswrapper[6014]: I1030 19:41:34.795371    6014 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5zffhh_34b518da-36c0-4652-b3bd-e1053a1226a3/istio-proxy/0.log"